Isometric Transformation Invariant and Equivariant Graph Convolutional Networks
Authors: Masanobu Horie
Naoki Morita
Toshiaki Hishinuma
Yu Ihara
Naoto Mitsume
Venue: ICLR 2021 Poster (2021)
Keywords: Machine Learning
Graph Neural Network
Invariance
Equivariance
Simulation
Mesh

Abstract: Graphs are one of the most important data structures for representing pairwise relations between objects. Specifically, a graph embedded in a Euclidean space is essential to solving real problems, such as physical simulations. A crucial requirement for applying graphs in Euclidean spaces to physical simulations is learning and inferring the isometric transformation invariant and equivariant features in a computationally efficient manner. In this paper, we propose a set of transformation invariant and equivariant models based on graph convolutional networks, called IsoGCNs. We demonstrate that the proposed model has a competitive performance compared to state-of-the-art methods on tasks related to geometrical and physical simulation data. Moreover, the proposed model can scale up to graphs with 1M vertices and conduct an inference faster than a conventional finite element analysis, which the existing equivariant models cannot achieve.

Published as a conference paper at ICLR 2021
ISOMETRIC TRANSFORMATION INVARIANT AND EQUIVARIANT GRAPH CONVOLUTIONAL NETWORKS
Masanobu Horie
Naoki Morita
University of Tsukuba,
University of Tsukuba,
Research Institute for Computational Science Co. Ltd. horietricos.
Research Institute for Computational Science Co. Ltd.
moritaêricos.cm ip
Toshiaki Hishinuma & Yu Ihara
Naoto Mitsume
Rescarch Institute for Computational Science Co. Ltd. {hi tnishmnuma, (hara))ricos co co.jp
University of Tsukuba
mit mitsume@kz tskubaa. ac.
ABSTRACT
Graphs are one of the most important data structures for representing pairwise re- lations between objects. Specifically, a graph embedded in a Euclidean space is essential to solving real problems, such as physical simulations. A crucial require- ment for applying graphs in Euclidean spaces to physical simulations is learning and inferring the isometric transformation invariant and equivariant features in a computationally efficient manner. In this paper, we propose set of transfor- mation invariant and equivariant models based on graph convolutional networks, called IsoGCNs. We demonstrate that the proposed model has a competitive per- formance compared state-of-the-as methods on tasks related geometrical and physical simulation data. Moreover, the proposed model can scale up to graphs with IM vertices and conduct an inference faster than a conventional finite ele- ment analysis, which the existing equivariant models cannot achieve.
INTRODUCTION
Graph-structured data embedded in Fuclidean spaces can be utilized in many different fields such as object detection, structural chemistry analysis, and physical simulations. Graph neural networks (GNNs) have been introduced to deal with such data. The crucial properties of GNNs include permutation invariance and equivariance. Besides permutations, isometric transformation invariance and equivariance must be addressed when considering graphs in Euclidean spaces because many properties of objects in the Euclidean space do not change under translation and rotation. Due to such invariance and equivariance, 1) the interpretation of the model is facilitated; 2) the output of the model stabilized and predictable; and 3) the training is rendered efficient by eliminating the necessity of data augmentation as discussed in the literature (Thomas et al. 2018 Weiler et al., 2018 Fuchs 2020).
Isometric transformation invariance and equivariance are inevitable, especially when applied to physical simulations, because every physical quantity and physical law either invariant or equiv- ariant to such a transformation. Another essential requirement for such applications is computational efficiency because the primary objective of learning physical simulation is replace computa- tionally expensive simulation method with faster machine learning model.
In the present paper, we propose IsoGCNs, a set simple yet powerful models that provide computationally-effiiiee isometric transformation invariance and equivariance based on graph con- volutional networks (GCNs) (Kipf & Welling 2017). Specifically, by simply tweaking the definition of an adjacency matrix, the proposed model can realize isometric transformation invariance. Because the proposed approach relies on graphs, it can deal with the complex shapes that are usually pre- sented using mesh point cloud data structures. Besides, specific form of the IsoGCN layer can be regarded as a spatial differential operator that is essential for describing physical laws. In addition, have shown that the proposed approach computationally efficient in terms of processing graphs

---

Published as a conference paper at ICLR 2021
with up to 1M vertices that often presented in real physical simulations. Moreover, the proposed model exhibited faster inference compared to a conventional finite element analysis approach at the same level of accuracy. Therefore, an IsoGCN can suitably replace physical simulations regarding its power to express physical laws and faster, scalable computation. The corresponding implemen- tation and the dataset available online"
The main contributions of the present paper can be summarized as follows: We construct isometric invariant and equivariant GCNs, called IsoGCNs for the specified input and output tensor ranks.
We demonstrate that IsoGCN model enjoys competitive performance against state-of- the-art baseline models on the considered tasks related to physical simulations.
We confirm that IsoGCNs are scalable to graphs with 1M vertices and achieve inference considerably faster than conventional finite element analysis.
RELATED WORK
Graph neural networks. The concept of a GNN was first proposed by Baskin et al. (1997); Sper- duti & Starita 1997) and then improved by (Gori et al. 2005; (Scasselil et al.] 2008). Although many variants of GNNs have been proposed, these models have been unified under the concept of mes- sage passing neural networks (Gilmer et al. 2017). Generally, message passing is computed with nonlinear neural networks, which can incur a tremendous computational cost. In contrast, the GCN developed by Kipf & Welling (2017) is considerable simplification of a GNN, that uses linear message passing scheme expressed as
Hout (AH W)
where H (Hout) is an input (output) feature of the Ith layer, Á is a renormalized adjacency matrix with self-loops, and W is a trainable weight. GCN, among the variants of GNNs, is essential to the present study because the proposed model is based on GCNs for computational efficiency.
X + Y is said be equivariant to a group G when [g z) = f(x), for all g E G and x E X, assuming that group G' acts on both X and Y. In particular, when flg x) = f(x). f is said be invariant to the group G. Group equivariant convolutional neural networks were first proposed by Cohen & Welling (016) for discrete groups. Subsequent studies have categorized such networks into continuous groups (Cohen et al. 2018). ditee-dimensionll data (Weiler et al. 2018), ) and general manifolds (Cohen et al. 2019).
Invariant and equivariant neural networks. A function
These methods are based on CNNs; thus, they cannot handle mesh or point cloud data structures as is. Specifically, 3D steerable CNNs (Weiler et al. 2018) uses voxels (regular grids), which though relatively easy to handle, are not efficient because they represent both occupied and non-occupied parts of object (Ahmed et al.] 2018). In addition, voxelized object tends to lose the smoothness of its shape, which can lead to drastically different behavior in physical simulation, as typically observed in structural analysis and computational fluid dynamics.
Thomas et al. (): Kondor (2018) discussed how to provide rotation equivariance to point clouds. Specifically. the tensor field network (TFN) (Thomas et al., (2018) is point cloud based rotation and translation equivariant neural network the layer of which can be written as
H
(1) out, = " H (
ΣV w (w, x;) Hinj
(k)
in,s
+
k20
k+1
(x) =
Σ
& ([
Σ
Yym (xmlx/1==/)]
(3)
J=|k-I]
m=-j
where (. (A (H, is a type-l input (output) feature at the éth vertex, o R2o - R is a trainable function, Yum is the mth component of the Jth spherical harmonics, and Qik is the Clebsch- Cordan coefficient. The SE(3)- Trans-Tranfr (Fuchs et al.) 2020) is a variant of the TFN with self- attention. These models achieve high expressibility based on spherical harmonics and message pass- ing with nonlinear neural networks. However, for this reason, considerable computational resources
https:: http://gttttttttee com 1owshi wttp://soggnniie httpoo//////////////11//111/111 r22001
2

---

Published as a conference paper at ICLR 2021
required. In contrast, the present study allows significant reduction in the computational costs because it eliminates spherical harmonics and nonlinear message passing. From this perspective, IsoGCNs are also regarded simplification of the TFN, seen in equation 14
Physical simulations using GNNs. Several related studies, including those by Saanhez-Gonzalez et al. [2018] 2019); Alet et al. (2019); Chang & Cheng (2020) focused applying GNNs to learn physical simulations. These approaches allowed the physical information to introduced to GNNs; however, addressing isometric transformation equivariance was out of the scope of their research.
In the present study, we incorporate isometric transformation invariance and equivariance into GCNs, thereby, ensuring the stability of the training and inference under isometric transformation. Moreover, the proposed approach is efficient in processing large graphs with up to 1M vertices that have a sufficient number of degrees of freedom to express complex shapes.
ISOMETRIC TRANSFORMATION INVARIANT AND EQUIVARIANT GRAPH CONVOLUTIONAL LAYERS
In this section, we discuss how to construct IsoGCN layers that correspond to the isometric invariant and equivariant GCN layers. To formulate model, assume that: 1) only attributes associated with vertices and not edges; and 2) graphs do not contain self-loops. Here, G = (V, E) denotes a graph and d denotes the dimension of a Euclidean space. In this paper, we refer to tensor as geomet- ric tensors, and we consider a (discrete) rank-p tensor field H(²) E RIV fxd"" where |/| denotes the number of vertices and f E Z (Z denotes the positive integers). Here, f denotes the number of features (channels) of H as shown Figure (a). With the indices, we denote H(p) kg... where i permutes under the permutation of vertices and ki,.... ..p refers to the Euclidean repre-
sentation. Thus, under the permutation, H(P) kiggkykk.... H(r) ka.... and under orthogonal transformation, H(p) kg..k + Elslerndd Ukit Ukatz ... Ukol, H(r) lighh
3.1 CONSTRUCTION OF ISOMETRIC ADIACENCY MATRIX
Before constructing an IsoGCN, an isometric adjacency matrix (IsoAM), which is the core of the IsoGCN concept must be defined. The proof of each proposition can be found in Appendix B
(a)
(b)
(1)
(1)
Gijus
4:2p
An IsoAM G IR VIx|Vixd is defined as:
Rd
G
Σ
(0 21), (4)
K.EVAKA
where Giju: is a slice in the spatial index of G,
2;; E RRd the position of the ith vertex (rank- I tensor), and Tijkt E pddd is an untrainable transformation invariant and orthogonal trans- formation equivariant rank-2 tensor. Note that we denote Gijck to be consistent with the no-
Figure 1: Schematic diagrams of (a) rank-1 ten-
sor field (()) with the number of features equal- ing 2 and (b) the simplest case of G.y. 5. [( - 2n) Aij(2,, a.:).
tation of H(P) kg..kk because and j permutes under the vertex permutation and k represents the spatial index while the number of features is always 1. The IsoAM can be viewed as a weighted ad- jacency matrix for each direction and reflects spatial information while the usual weighted adjacency matrix cannot because a graph has only one adjacency matrix. If the size of the set (Gıy:: # 0}₃ is greater than or equal to d, then it can be deemed to be a frame, which is a generalization of basis. For the simplest case, one can define Tij A I (Figure (b)), where ò,₃ is the Kro- necker delta, A is the adjacency matrix of the graph, and I is the identity matrix that is the simplest rank-2 tensor. In another case, Tijkı can be determined from the geometry of a graph, as defined in equation 16 Nevertheless, in the bulk of this section, forms of interaction, such as position-aware GNNs (You et al., 2019). Here, G is composed only untrainable parameters and thus be determined before training.
retain Tajkt abstract to cover various
3

---

Published as a conference paper at ICLR 2021
Proposition 3.1. IsoAM defined in equation 4 is both translation invariant and orthogonal rrans- formation equivariant, i.e., for any isometric transformation Vt RR,U U D(d).T:: Ux
T Giyak
Σ URI Gigisl
Based on the definition of the GCN layer in the equation
let G H E M denote the
convolution between G and the rank-O 0 tensor field H ER
vixf (f z+) as follows: (6)
(G H())
Digkk = Σ
G Gayak
H())
J:9:
With a rank-1 I tensor field H E RIVIsfxd let G ② H the contractions which defined as follows:
E V and GO G G denote
(G O
bi:: : Gejaa H
(G
Σ
Giyaa Gjuk-
Jaakk
J.k
J,k
The contraction of IsoAMs G @ G can be interpreted as the inner product of each component in the IsoAMs. Thus, the subsequent proposition follows.
Proposition 3.2. The contraction of IsoAMs G @ G is isometric transformation invariant, i.e., for
any isometric transformation vt E R E O(d), T: x Ux + G G G G.
With a rank-p tensor field (()) E IR V/x/xd2 let G ERIVix/xd*** and GO R denote the tensor products defined follows:
G & (())
Gejiat H(p) Jig: Jig:my mP.
(8)
L:ggkm mp
G
2 G) St:kyyk
Σ Gijykka G Gatakz
(9)
The tensor product of IsoAMs G @ G can be interpreted as the tensor product of each of the IsoAMs components. Thus, the subsequent proposition follows:
Proposition 3.3. The tensor product of the IsoAMs G & G is isometric transformation equivariant in terms of the rank-2 tensor, i.e., for any isometric transformation Vt E RR3 U E O(d), 4 Uw and Vi,j E 1,...: [V.. (G G) Gijkkkkk + UK,4, Ukgt2 (G
G) "igght
This proposition is easily generalized to the tensors of higher ranks by defining the pth tensor power
of G as follows:
X G = 1, G G. and X G
G x G. Namely, X G is isometric
transformation equivariant terms of rank-p tensor. Therefore, one can see that G H
G @ H()). Moreover, the convolution can be generalized for x and the rank-O tensor field (0) TR|V as follows:
(x"-1
p-1
G) &
(@a)
(0)
Σ
G
(0) H Jins
(10)
kg..
k2..k
The contraction can be generalized for x 2 G and the rank-q tensor field (9) V (p 2 4) specified below:
(
X о (())
Σ
X G
H Jig.n my...my...mmm...........
Kigll kg
M.
ijisks kg...kp mmmm.....
(11)
For the case p q. the contraction can be defined similarly.
3.2 CONSTRUCTION OF IsoGCN
Using the operations defined above, we can construct IsoGCN layers, which take the tensor field of any rank input, and output the tensor field of any rank, which differ from those of the input. In addition, one can show that these layers are also equivariant under the vertex permutation, as discussed in Maron al, (2018).
4

---

Published as a conference paper at ICLR 2021
3.2.1 ISOMETRIC TRANSFORMATION INVARIANT LAYER
As can be seen in Proposition 3.1 the contraction of IsoAMs isometric transformation invari- ant. Therefore, for the isometric transformation invariant layer with a rank-0 input tensor field Mm
H (0) H RRII (fin fout Z+), the activation function o, and the ((G G) H (
trainable parameter matrix W E Rfuxffou can be constructed as H Ho
By defining G G RIIXIV it can be simplified as H same form as a GCN (equation
10 LH W). which has the
D. with the exception that Á is replaced with L.
An isometric transformation invariant layer with the rank-p input tensor field H!!) E M can be formulated as Houl oo (H
([0' G (P)
w) If 1, such approaches
utilize the inner products of the vectors in Rd. these operations correspond to the extractions of relative distance and an angle of each pair of vertices, which are employed in Klicpera et al. (2020).
3.2.2 ISOMETRIC TRANSFORMATION EQUIVARIANT LAYER
To construct an isometric transformation equivariant layer, one can use linear transformation, con- volution and tensor product to the input tensors. If both the input and the output tensor ranks are greater than O, one can apply neither nonlinear activation nor bias addition because these opera- tions will cause an inappropriate distortion of the isometry because isometric transformation does not commute with them in general. However, a conversion that uses only linear transformation, convolution, and tensor product does not have nonlinearity, which limits the predictive performance of the model. To add nonlinearity to such a conversion, we can first convert the input tensors to rank-O ones, apply nonlinear activations, and then multiply them the higher rank tensors.
The nonlinear isometric transformation equivariant layer with the rank-m input tensor field H H(m)
R x x x and the rank-l (m < () output tensor H (t) RR [ fout xe! can be defined as:
out
[-m
Fm-+0 (H(m))
F Fm-si (H(m))
F Fin-i
(H()))
=
G
(m Hm
(12)
where x denotes multiplication with broadcasting and Wm E R fm foor are trainable weight ma-
trices multiplied in the feature direction. If m = O, we regard G <0 H (0)
as G
(()) If m I, one
can add the residual connection (He et al. 2016) in equation
If m > 4,
[m
out = F Fm-+0
(m))
x Fm-+/
(m)
F Fm-+i
(H!.
(m))
G
mm
(13)
In general, the nonlinear isometric transformation equivariant layer with the rank-0 to rank-M input tensor field (H (m)
(m) LM
Jm=u and the rank-l output tensor field ( can be defined as:
M
fgather (f ((FL (H())) M
① W
Σ m==0
H (((m))
(14)
k=
where fgather denotes a function such as summation, product and concatenation in the feature di- rection. One can see that this layer is similar to that in the TFN (equation 2 while there are no spherical harmonics and trainable message passing.
To be exact, the output of the layer defined above is translation invariant. To output translation equivariant variables such as the vertex positions after deformation (which change accordingly with the translation of the input graph), one first define the reference vertex position Tref for each graph, then compute the translation invariant output using equation [4] and finally, add Øref to the output. For more detailed information on IsoGCN modeling, Appendix
3.3 EXAMPLE OF ISOAM
The IsoGCN G is defined in a general form for the propositions to work with various classes of graph. In this section, we concretize the concept of the IsoAM to apply IsoGCN
5

---

Published as a conference paper at ICLR 2021
to mesh structured data. Here, mesh is regarded as graph regarding the points in the mesh as vertices of the graph and assuming two vertices are connected when they share the same cell. A concrete instance of IsoAM D,D E [e[V|[[Vix is defined as follows:
Difiik Dijink dig Σ Dilike
(15) Table 1: Correspondence between the differential operators and the expres- sions using the IsoAM D.
33 Di
Dijs: = M,
W/j Wij Aij(m),
(16)
|,, 2/
Differential op. Expression
M,
3t 3j
21 Di
Wal Au()), (17)
Gradient Divergence
D H (0)
| xi
[2| x
δ @ H
Laplacian
D @ DH
where R | 3 A(m) = min (- A 1) is an adjacency matrix up hops and Wly E IR is an un- trainable weight between the ith and jth vertices that determined depending on the tasks By regarding
Jacobian
D 2 H
Hessian
D @ D H
Tojtt 85052 M, Wii Aij(m)/|la;;
a; 1 in equation
one can see that D is qualified as an
IsoAM. Because a linear combination of IsoAMs is also an IsoAM, D is IsoAM. Thus, they provide both translation invariance and orthogonal transformation equivariance. D can be obtained only from the mesh geometry information, thus can be computed in the preprocessing step.
Here, D is designed such that it corresponds to the gradient operator model used in physical sim- ulations (Tamai & Koshizuka 2014| [wwrrz & Wendroff 1969). As presented in Table and and Ap- pendix C closely related to many differential operators, such as the gradient, divergence, Lapla- cian, Jacobian, and Hessian. Therefore, the considered IsoAM plays essential role in constructing neural network models that are capable of learning differential equations.
4 EXPERIMENT
To test the applicability of the proposed model, we composed the following two datasets: 1) a differential operator dataset of grid meshes; and 2) an anisotropic nonlinear heat equation dataset of meshes generated from CAD data. In this section, we discuss our machine learning model, the definition of the problem, and the results for each dataset.
Using D defined in Section 3.3 we constructed neural network model considering an encode- decoes configuration (Battaglia et al. 2018). The encoder and decoder were comprised of component-we MLPs and tensor operations. For each task, we tested m 2, equation T6 to investigate the effect of the number of hops considered. In addition to the GCN (Kipf & Welling
2017), we chose GIN (Xu al. 2018). SGCN (Wu et al.] (209)), Cluster-GCN (Chiang al. 2019)). and GCNII (Chen et al. 2020) GCN variant bascline models. For the equivariant models, chose the TFN (Thomas et al., 2018) and Traas-Transorr (Fuchs et al.) (020) as the baseline. We implemented these models using PyTorch 1.6.0 (Paszke al. 2019) and PyTorch Geometric 1.6.1 (Fey & Lenssen 2019). For both the TFN and (E(3))Transform
we used implementation of Fuchs et al. (2020) because the computation of the TFN is considerably faster than the original implementation, as claimed in Fuchs et al. (2020). For each experiment, we minimized the mean squared loss using the Adam optimizer (Kingma & Ba 2014). The corresponding implementa- tion and the dataset will be made available online. The details of the experiments can be found in
Appendix E and F
4.1 DIFFERENTIAL OPERATOR DATASET
To demonstrate the expressive power of IsoGCNs. we created dataset to learn the differential operators. We first generated a pseudo-2D grid mesh randomly with only one cell in the Z direction and 10 100 cells in the X and Y directions. We then generated scalar fields the grid meshes
2 M2 is invertible when the numher independent vectors { - xx. )1 is greater than or equal to the space dimension d, which is true for common meshes, c.g., a solid mesh in 3D Euclidean space. https: / thub. com http:/aaiaaaaaaaaaaaaaaaaa an

---

Published as a conference paper at ICLR 2021
1.66.01
Magint Magnhuus
31
3.06
3.0e000
Ground truth
-0.02
-5.e
Cluster-GCN GCN
IsoGCN (Ours)
Figure 2: (Top) the gradient field and (bottom) the error vector between the prediction and the ground truth of test data sample. The error vectors are exaggerated by a factor of 2 for clear visualization.
Table 2: Summary of the test losses (mean squared error + the standard error of the mean in the
original scale) of the differential operator dataset: 0 - 1 (the scalar field to the gradient field), 0 0+> (the scalar field the Hessian field), 1 + (the gradient field the Laplacian field), and 1 + 2 (the gradient field to the Hessian field). Here, if "xx" is "Yes", æ is also in the input feature.
We show only the best setting for each method except for the equivariant models. For full table, Appendix E
OOM denotes the out-of-memory on the applied GPU (32 GiB).
Loss of + 1 x10-5
Loss of 0 + 2
Loss of 1 + x10-6
Loss of + 2 x10-6
Method
hops
x10-6
GIN
5
Yes
147.07
0.51
47.35 1 0.35
404,92 + 1.74
46.18 0.39
GCNIT
5
151.13
0.53
31.87
+ 0.22
280.61 + 1.30
39.38
0.34
SGCN
5
Yes
151.16
0.53
55.08
0.42
127.21
0.63
56.97 0.44
GCN
5
Yes
151.14
0.53
48.50
0.35
542.30
= 2.14
25.37 0.28
Cluster-GCN
5
Yes
146.91
0.51
= 0.19
185.21
= 0.99
18.18 0.20
2
No
1 0.02
OOM
26.69 0.24
OOM
TFN
5
No
00M
OOM
0OM
OOM
2
No
1.79+ + 0.02
3.50 + 0.04
2.52 0.02
OOM
SB(3)-Trans.
5
No
2.12 +122:.02 0.02
00M
7.66 0.05
OOM
IsoGCN (Ours)
2
2.67 + 0.02
6.37 + 0.07
7.18 0.06
1.44 0.02 8.32 1 0.09
5
No
14.19 0.10
1..72.02.
34.09 0.19
and analytically calculated the gradient, Laplacian, and Hessian fields. We generated 100 samples for each train, validation, and test dataset. For simplicity, we set Wij = 1 in equation [6] for all (i,j) E. compare the performance with the GCN models, we simply replaced an IsoGCN layer with GCN its variant layers while keeping the number of hops m the same enable fair comparison. We adjusted the hyperparameters for the equivariant models to ensure that the number of parameters in each almost the same that in the IsoGCN model. For more details regarding the model architecture, see Appendix E We conducted the experiments using the following settings: 1) inputting the scalar field and predicting the gradient field (rank-0 rank-1 tensor); inputting the scalar field and predicting the Hessian field (rank-O - rank-2 tensor); 3) inputting the gradient field and predicting the Laplacian field (rank-1 - rank-0 tensor); and 4) inputting the gradient field and predicting the Hessian field (rank-1 - rank-2 tensor).
FigureQaam and Tableppresent a visualization and comparison of predictive performance, respectively. The results show that an IsoGCN outperforms other GCN models for all settings. This is because the IsoGCN model has information on the relative position of the adjacency vertices, and thus un- derstands the direction of the gradient, whereas the other GCN models cannot distinguish where the adjacencies are, making nearly impossible to predict the gradient directions. Adding the vertex po- sitions to the input feature to other GCN models exhibited a performance improvement, however as the vertex position is not a translation invariant feature, it could degrade the predictive performance of the models. Thus, we did not input ariant models to retain their isometric transformation invariant and equivariant natures. IsoGCNs perform competitively against other equivariant models with shorter inference time shown in Ta-
vertex feature to the IsoGCN model other equiv-

---

Published as a conference paper at ICLR 2021
0.6
02
02
De-00
Ground truth
5.0e-01
Inference Cround Trurs
D2
-0.2
+5.0-
SGCN
SE(3) Trans.
IsoGCN (Ours)
Figure 3: (Top) the temperature field of the ground truth and inference results and (bottom) the error between the prediction and the ground truth of a test data sample. The error is exaggerated by factor of 2 for clear visualization.
ble 7 As mentioned Section 3.3 D corresponds to the gradient operator, which is now confirmed
in practice.
4.2 ANISOTROPIC NONLINEAR HEAT EQUATION DATASET
To apply the proposed model to real problem, we adopted the anisotropic nonlinear heat equation. We considered the task of predicting the time evolution of the temperature field based on the ini- tial temperature field, material property, and mesh geometry information as inputs. We randomly selected 82 CAD shapes from the first 200 shapes of the ABC dataset (Koch et al., 2019). generate first-order tetrahedral meshes using mesh generator program, Gmsh (Geuzaine & Remacle 2009), ) randomly set the initial temperature and anisotropic thermal conductivity, and finally conducted finite element analysis (FEA) using the FEA program FrontISTR (Morita et al. 2016 Ihara al. 2017).
For this task, the ith vertex (equation 46) Similarly to the differential operator dataset, we tested the number of hops m 2, 5. However because put four IsoAM operations in model, the number of hops visible from the model is 8 (m 2) or 20 (m = 5). As is the case with the differential operator dataset, we replaced an IsoGCN layer accordingly for GCN or its variant models. In the case of k = 2, we reduced the number of parameters for each of the equivariant models to fewer than the IsoGCN model because they exceeded the memory of the GPU (NVIDIA Tesla V100 with GiB memory) with the same number of parameters. In the case of k = 5, neither the TFN nor the SE(3)- Transformer fits into the memory of the GPU even with number parameters equal to 10. For more details about the dataset and the model, see Appendix F
way Preffective / / /Veffective where Veffective denotes the effective volume of
Figure 3 and Table present the results of the qualitative and quantitative comparisons for the test dataset. The IsoGCN demonstrably outperforms all other baseline models. Moreover, owing the computationally efficient isometric transformation invariant nature IsoGCNs, also achieved a high prediction performance for the meshes that had a significantly larger graph than those con- sidered in the training dataset. The IsoGCN can scale up to 1M vertices, which practical and is considerably greater than that reported in Gonzalez et al. (2020). Therefore, we conclude that IsoGCN models can be trained on relatively smaller mesheş to save the training time and then used apply the inference larger meshes without observing significant performance deterioration.
Table reports the preprocessing and inference computation time using the equivariant models with m the number hops and FEA using FronUSTR 5.0.0. We varied the time step (At
applied private update to FrontISTR to deal with the anisotropic heat problem, which will be also made available online. However, it should also be sufficiently large express sample shapes and fields.
https: / /i thub. com/Front STR/ rentt
rent ISTR
8

---

Published as a conference paper at ICLR 2021
1.0,0.5) 0.5) for the FEA computation to compute the t 1.0 time evolution thus, resulting in different computation times and errors compared to an FEA with At = 0.01, which was considered as the ground truth. Clearly, the IsoGCN is 3- to 5- times faster than the FEA with the same level of accuracy, while other equivariant models have almost the same speed as FrontITT with At = 0.5.
5 CONCLUSION
In this study, we proposed the GCN- based isometric transformation in- variant and equivariant models called IsoGCN. We discussed exam- ple of an isometric adjacency ma- trix (IsoAM) that was closely re- lated to the essential differential op- erators. The experiment results con- firmed that the proposed model lever- aged the spatial structures and can deal with large-scale graphs. The computation time of the IsoGCN model is significantly shorter than the FEA, which other equivariant models cannot achieve. Therefore, IsoGCN must be the first choice to learn phys- ical simulations because of its com- putational efficiency as well as iso- metric transformation invariance and equivariance. Our demonstrations were conducted on mesh struc- tured dataset based on the FEA re- sults. However, we expect IsoGCNs to be applied to various domains, such as object detection, molecular property prediction, and physical simulations using particles.
Table 3: Summary of the test losses (mean squared error + the standard error of the mean in the original scale) of the anisotropic nonlinear heat dataset. Here, if "x" is "Yes", is also the input feature. We show only the best setting for each method except for the equivariant models. For the full table, see Appendix E OOM denotes the out-of- out-of-memory on the applied GPU (32 GiB).
Loss
Method
# hops
x10
GIN
2
No
16.921 = 0.040
GCN
No
10.427 = 0.028
GCNII
5
No
8.377 = 0.024
Gluster-GCN
2
No
7.266 + 0.021
SGCN
5
No
6.426 + 0.018
2
No
15.661 + 0.019
TFN
No
OOM
2
No
14.164 + 0.018
SB(3)-Trans.
5
No
OOM
2
No
4.674 + 0.014
IsoGCN (Ours)
5
No
2.470 = 0.008
ACKNOWLEDGMENTS
The authors gratefully acknowledge Takanori Maehara for his helpful advice and NVIDIA for hard- ware donations. This work was supported by JSPS KAKENHI Grant Number 19H01098.
Table 4: Comparison computation time. To generate the test data, sampled CAD data from the test dataset and then generated the mesh for the graph to expand while retaining the element volume almost the same size. The initial temperature field and the material properties are set randomly using the same methodology each computation was run on the same CPU (Intel Xcon E5-2695 v2@2.40GHz) using one core, and we excluded file I/O time from the measured time. OOM denotes the out-of-memory (500 GiB).
the dataset sample generation. For fair comparison,
| 21, 289
VV = 155, 019
| = 1, 011, 301
Loss
Loss x x10-4
Loss X 10-4
Method
Time [s]
Time [s]
Time [s]
x x10-4
FrontISTR (At 1.0)
10.9
16.7
6.1
181.7
2.9
1656.5
FronuSTR (At 0.5)
0.8
30.5
0.4
288.0
0.2
2884.2
TFN
77.9
46.1
30.1
400.9
OOM
OOM
111.4
31.2
80.3
271.1
OOM
0OM
IsoGCN (Ours)
8.1
7.4
4.9
84.1
3.9
648.4

---

Published as a conference paper at ICLR 2021
REFERENCES
Eman Ahmed, Alexandre Saint, Abd El Rahman Shabayek, Kseniya Cherenkova, Rig Das, Gleb Gusev, Djamila Aouada, and Bjom Ottersten. A survey on deep learning advances on different 3d data representations. arXiv preprint arXiv: (008001462,0000
Ferran Alet, Adarsh Keshav Jeewajee, Maria Bauza Villalonga, Alberto Rodriguez, Tomas Lozano- Perez, and Leslie Kaelbling. Graph element networks: adaptive, structured computation and memory. In ICML, 2019.
Igor Baskin, Vladimir A Palyulin, and Nikolai S Zefirov. A neural device for searching direct correlations between structures and properties of chemical compounds. Journal of chemical in- formation and computer sciences, 37(4):715-721, 1997.
Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint 1r0iv:1806.0111 2018.
Vinicius Zambaldi,
Kai-Hung Chang and Chin-Yi Cheng. Learning to simulate and design for structural engineering. arXiv preprint arXiv:2003.09100 2020.
Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks, arXiv preprint arXiv:2007.02/:6 003,03...33 2020.
Wei-Lin Chiang, Xuanqing Liu, Si Si, Yang Li, Samy Bengio, and Cho-Jui Hsieh, Cluster-gen: An efficient algorithm for training deep and large graph convolutional networks. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 257-266, 2019.
Taco Cohen and Max Welling. Group equivariant convolutional networks. In International confer- ence on machine learning, pp. 2990-2999, 2016.
Taco S Cohen, Mario Geiger, Jonas Köhler, and Max Welling. Spherical cnns. In ICLR, 2018.
Taco S Cohen, Maurice Weiler, Berkay Kicanaoglu, and Max Welling, Gauge equivariant convolu- tional networks and the icosahedral cnn. ICMI., 2019.
Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019.
Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling. Se transsfsffoss:ss: 3d roto- translation equivariant attention networks. Advances in Neural Information Processing Systems, 33, 2020.
Christophe Geuzaine and Jean-François Remacle. Gmsh: a three-dimensional finite element mesh generator with built-in pre- and post posscssiccssng facilities. International Journal for Numerical Methods in Engineering. 79(11): 1311)))139-1131 1331, 2009.
Justin Gilmer, Samuel Schoenholz, Patrick Riley, Oriol Vinyals, and George Dahl. Neural message passing for quantum chemistry. In Proceedings of the 34th International Conference on Machine Learning-Vollen Volune 70, pp. 1263-1272. JMLR. org, 2017.
Marco Gori, Gabriele Monfardini, and Franco Scarselli. A new model for learning in graph domains. In Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005., volume 2, pp. 729-734. IEFF, 2005.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Yu Ihara, Gaku Hashimoto, and Hiroshi Okuda. Web-based integrated cloud cae platform for large- scale finite element analysis. Mechanical Engineering Letters, 3:17-00520, 2017.
10

---

Published as a conference paper at ICLR 2021
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6990, 2014.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. In ICLR, 2017.
Johannes Klicpera, Janek Groß, and Stephan Günnemann. Directional message passing for molec- ular graphs. In ICLR, 2020.
Sebastian
Koch, Albert Matveev, Zhongshi Jiang, Francis Williams, Alexey Artemov, Evgeny Bur- naev, Mare Alexa, Denis Zorin, and Daniele Panozzo. Abe: A big cad model dataset for geometric deep learning. In The IEEE Conference on Computer Vision and Puttern Recognition (CVPR),
June 2019.
Risi Kondor.
N-body networks: a covariant hierarchical neural network architecture for learning atomic potentials. arXiv preprint (rXXiv:8033.01 2018.
Haggai Maron, Heli Ben-Hamu, Nadav Shamir, and Yaron Lipman. Invariant and equivariant graph networks. arXiv preprint arXiv: 18/2.09902, 2018.
Naoki Morita, Kazuo Yonekura, Ichiro Yasuzumi, Mitsuyoshi Tsunori, Gaku Hashimoto, and Hi- roshi Okuda. Development of x 3 dof blocking structural elements to enhance the computational intensity of iterative linear solver. Mechanical Engineering Letters, 2:16-00082, 2016.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-IO), pp. 807-814, 2010.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary De Vito,, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. H. Wallach, H. Larochelle, Beygelzimer, F. d' d'Alché- Buce E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019.
Alvaro Sanchez-Gonzaler Nicolas Heess, Jost Tobias Springenberg. Josh Merel, Martin Riedmiller, Raia Hadsell, and Peter Battaglia. Graph networks as learnable physics engines for inference and control. arXiv preprint arXiv:1806. 01242. 2018.
Alvaro Sanchez-Gonzalez,, Victor Bapst, Kyle Cranmer, and Peter Battaglia. Hamiltonian graph networks with ode integrators. arXiv preprint arXiv:1909.12740 2019.
Alvaro Sanchez-Gonzalez Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Pe- ter W Battaglia. Learning simulate complex physics with graph networks. arXiv preprint arXiv:2002.09900. 2020.
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE Transactions on Neural Networks, 20( 20(1):61-80, 2008.
Alessandro Sperduti and Antonina Starita. Supervised neural networks for the classification of structures. IEEE Transactions on Neural Networks, 8(3):714-735, 1997.
Blair Swartz and Burton Wendroff. Generalized diffe-eifference schemes. Mathematics of Compu- tation, 23(105):37-49, 1969.
Tasuku Tamai and Seichhi Koshizuka. Least squares moving particle semi-implicit method. Com- putational Particle Mechanics, 1(3):277-305, 2014.
Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-and translation-equie eequiaarinn-uuvvriiitt neural networks for 3d point clouds. arXiv preprint arXiv: 180..829,, 2018.

---

Published as a conference paper at ICLR 2021
Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco S Cohen. 3d steerable cnns: Learning rotationally equivariant features in volumetric data. In NewrlPS, pp. 10381-10392 2018.
Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. Simpli- fying graph convolutional networks. In ICML, pp. 6861-6871. PMLR, 2019.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint arXiv:1810.00826. 2018.
Jiaxuan You, Rex Ying, and Jure Leskovee. Position-aware graph neural networks. arXiv preprint arXiv:1906.0450 04817, 2019.
A NOTATION
A graph A vertex set The number of vertices
|
An edge set
The positive integers
d
The dimension of the Euclidean space
a4
The position of the ith vertex Flement of a;;
G E IR xd
The isometric adjacency matrix (IsoAM) (equation 4
G Jis E Rd
Slice of G in the spatial index (equation 4
Gojjkk E R
Element (i,j. k) of
H(P) E /
A rank-p tensor field tensor (/,p E Z+)
H(P)
.........p)) Element (i: k1, .., kp)) of H(P), i refers to the permutation rep- resentation, ki.... kp refer to the Euclidean representation, and de- notes the feature index (See section 3).
1:gi/s Kpp
G H(()
Convolution of the pth power of G and rank-0 tensor field ( (equa- tion 6 equation 10
(())
Contraction of the pth power of G and rank-q tensor fields (equation 7 equation
G
X ((9)
Tensor product of the pth power of G and rank-q tensor fields H (4) (equation X)
The rank-p input tensor field of the considered layer
mn
[p)
The rank-p output tensor field of the considered layer
The activation function
W
The trainable parameter matrix
A E [
An adjacency matrix
85
The Kronecker delta
V roffective
The effective volume of the ith vertex (equation 46
V. ymean
The mean volume of the ith vertex (equation 47
D R X
A concrete instance of IsoAM (equation 15
12

---

Published as a conference paper at ICLR 2021
B PROOFS OF PROPOSITIONS
In this section, we present the proofs of the propositions described in Section
Let
R 9 g 23)) (æx æz). Note that expressed using g( as djss ExIeV Tijkig [1. vt).
B. PROOF OF PROPOSITION
Proof. First, demonstrate the invariance with respect the translation with At E R g( Dz)) is transformed invariantly as follows under translation:
g + t, k. +
+) = [ + = (x; + +]]
g(; Tz)).
(18)
By definition, Tiski is also translation invariant. Thus,
(a + t, Db + t)
Σ Negh
Tasig xxk)
A.EVA
Gija-
(19)
We then show equivariance regarding the orthogonal transformation with VU O(d). g( 23)) transformed as follows by orthogonal transformation:
g(Ux, Uz,) Ux Uv
U(z, xii)
Ug( x.y)-
(20)
By definition, Tisks is transformed to UT UT U by orthogonal transformation. Thus,
Σ
UT.HU-1g(U Uxx)= = Σ UTINU-UUU(R),.
KEEVAK
K.
UGG
(21)
Therefore, G is both translation invariant and an orthogonal transformation equivariant.
B.2 PROOF PROPOSITION 3.2
Proof. Here, G @ G is translation invariant because G is translation invariant. We prove rotation invariance under an orthogonal transformation YU E On)). In addition, Go G is transformed under U follows:
Σ Gijikk Gjtck
Σ
Ukm Gigam Ukn Gfla..
i.k,m,
Σ
Ukm km Gijiom Gjtin
m.
Σ J.
UHh Ukn Gijiim Gittin
Σ 8mn Gijiim Gytan
( property of the orthogonal matrix)
Σ Giggim Gytyo
Σ
Giysk Gytük-
( Change the dummy index m -> k)
(22)
J.k
Therefore, G ④ G is isometrie transformation invariant.
13

---

Published as a conference paper at ICLR 2021
B.3 PROOF OF PROPOSITION 3.3
Proof. G G is transformed under VU E (()) as follows:
Σ Geiik Gplism H
Σ
Un Gajin Umo Gilloo
n,0
Σ
Ukn Gijai G om.
(23)
n,
By regarding Gijan Gytno one matrix Hno. it follows the coordinate transformation of rank-2 tensor U UHU for each j. and l.
C PHYSICAL INTUITION OF D
In this section, we discuss the connection between the concrete IsoAM example and the dif- ferential operators such the gradient, divergence, the Laplacian, the Jacobian, and the Hessian operators.
Let o, denote rank-0 tensor (scalar) the ith vertex. Let assume partial derivative model of rank-0 tensor & at the ith vertex regarding the kth axis (06/Ozz) E R (A: E {l....,de .d)). that based on the gradient model in the least squares moving particle semi-implicit method (Tamai & Koshizuka 2014).
OΦ
Φj Øj
Fijk
Fik x.,
Wij Aiy
(24)
0rk
| xzl]
Σ D Dos - -s))
(25)
M,
Σ
2t 3i
Wi Wa (mn)
(26)
xt x;
D;
Although could define wi function of the distance ||, - x2 |. way kept constant with respect to the distance required to maintain the simplicity of the model with fewer hyperparameters.
C.1 GRADIENT
D can be viewed Laplacian matrix based D; however, H can be interpreted as the gradient within the Euclidean space. Let ( E / be an approximation of the gradient of (0) Using equation 25 the gradient model can be expressed as follows:
(0
H
(27)
k:gok
Oxk
Diyxe (((0)
-H00)
(28)
14

---

Published as a conference paper at ICLR 2021
Using this gradient model, confirm that (D (()) ()HH) ligik (V H(O))
Fig because
[ * 0() kinkk
Σ Disish H
(29)
5:9:
[Diiiu dig D Dujk H
Duiii Dasak Po) 3:9:
Σ 11
Beg Ditgh j:s
Σ Disnk Disik J:0:
Σ Dupu H0))
Σ Dijak H(O)
DisssH
Σ Dijsk H())
(:' Change the dummy index - j)
3:9:
kg:
Σ D (H
Heg
3:0:
(())
(30)
kgik
Therefore, D* can be interpreted as the gradient operator within a Euclidean space.
C.2 DIVERGENCE
We show that D (()) corresponds the divergence. Using D, the divergence model V H())
RIV) x is expressed follows:
Σ 2 (1)
V (V.HO))
(31)
Ozk
1::: Dajink (H(1)
Σ
H, High)
(32)
Jigch
d.ke
Then, D @ (())
D H (1)
Σ
D
Uick (())
lig:
Eigst
j.k
Σ
D Dijich
diz
Σ
H(1) Figs
3.k
D
(1)
Σ D
Dit
H
(1) sink
Jigsk
j.k
1.k
Σ
Disssh (H(1)
High
H(a)
Change the dummy index j)
j.k
(V
H())
Dig:"
(33)
C.3 LAPLACIAN OPERATOR
We prove that D D corresponds to the Laplacian operator within Fuclidean space.
15

---

Published as a conference paper at ICLR 2021
Using equation 25
the Laplacian model 2 H
RMXX can be expressed follows:
0H
V2
(())
ki::
dok
Oox
2-
ØH
ÖH
Σ Dijish
Oork
j.k
jia:
Diijit Ligak
Dgnns Daickl b:g
(
Σ
Ditsh (H -
Desite DRitk
D (H
(34)
l:g:
Then, (D () is
((D D)
Σ
Dinck Se
H lgg:
::g:
S.k.
(DArAk 85t DDink
Dijak dig Σ
Domai
Dinck H
k.k.
m7
D Siink Drich HLo
Dijuh D Dinakh 4,0))
j.k.
J.k.n
D Domen Dilik
H
H100)
Σ
D Damass Dinck
((0)
h.l.m
k,m,n
Disiih St
Σ
Disssh D Drncc (())
l:g:
3:g:
k.k.
j.h.n
Σ
Dinn D uakk
(0)
+
Σ
Disik D H
ligs
kdd
(:: Change the dummy index m + j for the third and fourth terms)
Σ
Dennk (Djt.kk Du. (0)
Lg: - H000
J.k.
(t: Change the dummy index - I for the second and fourth terms)
= (v' 0())
(35)
C.4 JACOBIAN AND HESSIAN OPERATORS
Considering similar discussion, we can show the following dependencies. For the Jacobian model, J[ IR /xxxxx
2 (1)
(JHHS) konk!
(36)
Ox
kigg
Σ Dijut (1)
H())
(37)
Jigh
(D & Vigi
(38)
16

---

Published as a conference paper at ICLR 2021
For the Hessian model, Hess (
E R
(Hess] ()))
(
(39)
Eigkt
k doa
Eg:
Σ
Dajinkl Smer (H())
H( D Demest (H
H H
(40)
J.m
(())
(41)
tighll
D IsOGCN MODELING DETAILS
To achieve isometric transformation invariance and equivariance, there are several rules follow. Here, we describe the desired focus when constructing an IsoGCN model. In this section, rank-p tensor denotes tensor rank of which is 2 and a denotes a nonlinear activation function. is a trainable weight matrix and b is trainable bias.
D.1 ACTIVATION AND BTAS
As the nonlinear activation function is not isometric transformation equivariant, nonlinear activation to rank-p tensors cannot be applied, while one can apply any activation to rank-O tensors. In addition, cannot add bias when performing
adding bias is also not isometric transformation equivariant, an affine transformation to rank-p tensors. Again, one can add bias to rank-0 tensors.
Thus, for instance, if one converts from rank-O tensors (0) to rank-1 1 tensors using IsoAM G, Geo (H W+b) and (G*o(H (H W are isometric equivariant functions, however (G+H ) W+b
and a (G o (H W) are not due to the bias and the nonlinear activation, respectively. Like-
wise, regarding conversion from rank-1 tensors H (1) to rank-0 tensors, (G (()) W + +b) and
( (H w)) isometric transformation invariant functions; however, G (H W + b)
and o (HOHH)) W + b are not.
To convert rank-p tensors to rank-q tensors (g 1), apply neither bias nor nonlinear tivation. To add nonlinearity to such a conversion, we can multiply the converted rank-O tensors
((x G e (()) W + b) with the input tensors (p) or the output tensors H(9)
D.2 PREPROCESSING OF INPUT FEATURE
Similarly to the discussion regarding the biases, we have to take care of the preprocessing of rank-p tensors retain isometric transformation invariance because adding a constant array and component-wise scaling could distort the tensors, resulting in broken isometric transformation cquivariance.
For instance, H(²) (im)/tdall
H (p)
is valid transformation retain isometric transformation equiv-
ariance, assuming Stdau
(P) E is standard deviation of all components of ((r) However,
conversions such as (P)
(P)) and ((P) Mcan
(P)) are not isometric trans-
/Stdeomponent
formation equivariant, assuming that Stdcomponent (P)) IRd" is a component-wise standard deviation.
D.3 SCALING
Because the concrete instance of IsoAM D corresponds to the differential operator, the scale of the output after operations regarding H can be huge. Thus, rescale D using the scaling factor
17

---

Published as a conference paper at ICLR 2021
1/2
Mcan (D s6;1 and vertices.
where saapee, denotes the mean over the samples
D.4 IMPLEMENTATION
Because an adjacency matrix A is usually a sparse matrix for a regular mesh, A(m) in equation 16 also sparse matrix for sufficiently small Thus, we can leverage sparse matrix multiplica- tion in the IsoGCN computation. This is one major reason why IsoGNNs can compute rapidly. the multiplication (tensor product contraction) of IsoAMs must be computed multiple times the associative property of the IsoAM can be utilized.
For instance, is apparent that G *
G (G.... (G) (G.... Assuming that the number
of nonzero elements in (mm) equals and H E g|ixx, then the computational complexity of the right-hand side Orn()]fd'*) This an exponential order regarding However, d and usually small numbers (typically d 3 and k I 4). Therefore layer with a realistic spatial dimension d and tensor rank k fast and memory efficiently. our implementation, both sparse matrix operation and associative property utilized to realize fast computation.
compute an IsoGCN
EXPERIMENT DETAILS: DIFFERENTIAL OPERATOR DATASET
E.1 MODEL ARCHITECTURES
(a)
MLP 8, tanh, tanls
BoGGN D [64,64] [kdentity]
[64, I]
V
[Identity]
(b)
MLP [1. 8, 64]
IN D & D @ [61.64]
Lineez 644, 1] Identity:
V
X
V
"tanh, tank]
[Identity]
(c)
LooGCN
MLP (61,,11, 61,6666]) [tanb, tanb, Identity]
164, 64] [cbentit]]
Linear Lddentity
laGCN D 64, 64,64] [dentity:
Linerr [64.
226
Identity)
(d)
IsOGNN D 164, 64] [edentit]]
MLP (64, [tanh, tanh, Idlentty]]
boGCN [[dettiy]]
Linear [1,61 [vimiityy
Lanear G4, [dentity
V
V
Encoder
Propagation
Decoder
Figure 4: The IsoGCN model used for (a) the scalar field to the gradient field, (b) the scalar field to the Hessian field, (c) the gradient field the Laplacian field, (d) the gradient field the Hessian field of the gradient operator dataset. The numbers in each box denote the number of units. Below the unit numbers, the activation function used for each layer is also shown. X denotes the multiplication in the feature direction.
18

---

Published as a conference paper at ICLR 2021
Table 5: Summary of the hyperparameter setting for both the TFN and StanoTrranformrr. For the parameters not in the table, we used the default setting in the implementation of https:// public
0 1
0 0 2 1 0
1
2
hidden layers
NL layers in the self-interaction channels
24
20
24
24
maximum rank of the hidden layers
# nodes in the radial function
16
16
22
Figure represents the IsoGCN model used for the differential operator dataset. We used the lanh activation function as nonlinear activation function because we expect the target temperature field be smooth. Therefore, we avoid using non-differenntiable activation functions such as the rectified linear unit (ReLU) (Nar & Hinton 2010). For GCN and its variants, we simply replaced the IsoGCN layers with the corresponding ones. We stacked m (= 2,5) layers for GCN, GIN, GCNII, and Cluster-GCN. We used an m hop adjacency matrix for SGCN.
For the TFN and SE(3)-Transffrmer Transformer, we set the hyperparameters have almost the same number of parameters as in the IsoGCN model. The settings of the hyperparameters are shown in Table 5
E.2 RESULT DETAILS
Table rrpresents the detailed comparison of training results. The results show that an IsoGCN outperforms other GCN models for all settings. Compared to other equivariant models, IsoGCN has competitive performance compared to equivariant models with shorter inference time as shown in Table 7 Therefore, it be found out the proposed model has strong expressive power to press differential regarding space with less computation resources compared to the TFN and SE(3)- Transformer.
F EXPERIMENTS DETAILS: ANISOTROPIC NONLINEAR HEAT EQUATION DATASET
F.1 DATASET
The purpose of the experiment was solve the anisotropic nonlinear heat diffusion under an adia- batic boundary condition. The governing equation is defined as follows:
Ω R
(42)
(T(x,t) V ()
(43)
Ø1
T(x, 1. 0) (ax) in Ω
(44)
VT(x, mmm, n((r)) O, on an,
(45)
where 7' is the temperature field, To.o is the initial temperature field, C' E [Rdxd is an anisotropic
diffusion tensor and n²x;;) the normal vector 26 E 8Ω. Here, C depends on temperature
thus the equation is nonlinear. We randomly generate C(T = -1 for it to be positive semidef- inite symmetric tensor with eigenvalues varying from 0.0 to 0.02. Then, we defined the linear temperature dependency the slope of which is -C(T diffusion tensor is uniform for each sample. The task defined to predict the temperature field at t 0.2, 0.4, 0.6, 1.0 (To.2, To. To.o, To.s. Ti.o) from the given initial temperature field, material property, and mesh geometry. However, the performance evaluated only with Tioo o to focus on the predictive performance. We inserted other output features to stabilize the trainings. Accordingly, the diffusion number of this problem is CAt/ ~ 10.0* assuming Ar ~ 10.0-3
11/4. The function of the anisotropic
Figure 5 represents the process of generating the dataset. We generated up 9 FEA results for each CAD shape. To avoid data leakage in terms of the CAD shapes, we first split them into training, validation, and test datasets, and then applied the following process.
19

---

Published as a conference paper at ICLR 2021
Table 6; Summary of the test losses (mean squared error + the standard error of the mean in the
original scale) of the differential operator dataset: 0 - 1 (the scalar field to the gradient field), (the scalar field the Hessian field), - (the gradient field the Laplacian field), and
1 2 (the gradient field to the Hessian field). Here, if "w" is "Yes", is also in the input feature.
Loss of +
Loss of 0 + x10-6
Loss of 1 +
Loss
2
Method
# hops
x10-6
x10
x10-6
2
No
151.19
0.53
49.10 = 0.36
542.52 + 2.14
59.65 0,46
2
Yes
147.10
0.51
47.56 + 0.35
463.79 + 2.08
50.73 0.40
GIN
5
No
151.18
0.53
48.99 + 0.36
542.54 + 2.14
59.64 0.46
5
Yes
147.07
0.51
+7.550..3
404.92 + 1.74
46.18 0.39
2
No
151,18 = 0.53
43.08
542.74 = 2,14
59,65
0.46
2
Yes
151.14 = 0.53
40.72
194.65 1 1.00
45.43 0.36
GCNII
5
No
151.11 0.53
32.85
542.65 2.14
59.66 0.46
5
Yes
151.13 0.53
31.87
280.61 1.30
39.38 0.34
2
No
151.17 0.53
$42.90 + 2.14
59.65 0.46
2
Yes
151.12 = 0.53
49.96
353.29 = 1.49
59,61 0.46
SGCN
5
No
151.12 + 0.53
+
542.73 + 2.14
59.64 0.46
5
Yes
151.16 + 0.53
55.08 0
127.21 + 0.63
56.97 0.44
151.23 0.53
49.59 0.33
542.54 2.14
59.64
0.46
2
Yes
151.14
0.53
47.91 0.35
542.68 + 2.14
59.60 0.46
GCN
5
No
151.18 0.53
50.58 0.38
542.53 2.14
59.64 0.46
5
Yes
151.14 0.53
48.50 + 0.35
542.30 2.14
25.37 0.28
2
151.19
0.53
33.39 0.24
$42.54
59.66 0.46
2
Yes
32.29 - 0.24
167.73
0.83
17.72 0.17
Cluster-GCN
5
No
151.15
0.53
28.79 = 0.21
542.51 = 2.14
59.66
0.46
5
Yes
146.91
0.51
26.60 + 0.19
185.21 + 0.99
18.18 0.20
2
No
2.47 0.02
OOM
26.69 0.24
OOM
TFN
5
OOM
OOM
OOM
2
No
1,79 + 0.02
3,50 + 0.04
2.52 0.02
OOM
SE(3)-Tram)
5
No
2.12 + 0.02
0OM
7.66 + 0.05
OOM
2
No
2.67 + 0.02
6.37 0.07
7.18 + 0.06
1.44 + 0.02
IsoGCN (Ours)
5
14.19
21.72 + 0.25
34.09 0.19
8.32 0.09
Table Summary of the inference time on the test dataset. corresponds to the scalar field to the gradient field, and 0 - 2 corresponds to the scalar field to the Hessian field. Each computation was run on the same GPU (NVIDIA Tesla V100 with 32 GiB memory). OOM denotes the out-of- memory of the GPU.
- 1
+ 2
Method
# parameters
Inference time
# parameters
Inference time
TFN
5264
3.8
5220
0OM
SB(3)-Trans.
5392
4.0
5265
9.2
IsoGCN (Ours)
4816
0.4
4816
0.7
Using one CAD shape, we generated to three meshes using clscale (a control parameter of the mesh characteristic lengths) 0.20, 0.25, and 0.30. To facilitate the training process, we scaled meshes to fit into a cube with edge length equal to 1.
Using one mesh, we generated three initial conditions randomly using a Fourier series of the 2nd to 10th orders. We then applied an FEA to each initial condition and material property determined randomly as described above. We applied an implicit method solve time evolutions and direct method to solve the linear equations. The FEA time step At was set to 0.01.
During this process, some the meshes or FEA results may not have been available due to excessive computation time converegeee Therefore, the size of the dataset was not exactly equal to the number multiplied by 9. Finally, we obtained 439 FEA results for the training dataset, 143 FEA results for the validation dataset, and 140 FEA results for the test dataset.
20

---

Published as a conference paper at ICLR 2021
Initial condition
CAD
Mesh
Material property
FEA results
clscale = 0.30
clscale 0.25
elscale 0.20
IEMPEEATURE -0.6-04-02
-1.06+00
0.2 044 0.6 1.0e+00
Figure 5: The process of generating the dataset. smaller clscale parameter generates smaller meshes.
F.2 INPUT FEATURES
To express the geometry information, we extracted the effective volume of the ith vertex Veffective and the mean volume of the ith vertex Vimann, which are defined follows:
veffective
Σ
2 Ve.
(46)
4
RENT
EcENs V
mean
(47)
W
where NO is the set of elements, including the ith vertex.
For GCN its variant models, tested several combinations of input vertex features To.o, C. Peffective ymean, and æ (Table 9). For the IsoGCN model, inputs were To.0- C, Veffective and /mmaa
F.3 MODEL ARCHITECTURES
Secure6/rppreeettt the IsoGCN model used for the anisotropic nonlinear heat equation dataset. We used the tanh activation function a nonlinear activation function because we expect the target tem- perature field to be smooth. Therefore, we avoid using non-differentiable activation functions such as the rectified linear unit (ReLU) (Nair & Hinton 2010). Although the model looks complicated, one
propagation block corresponds to the first-order Taylor expansion T(t At) vCo +T())+((I
because the propagation block is expressed D C C MLP( D T. where T denotes the
21

---

Published as a conference paper at ICLR 2021
[32,511 [denets]
Yio Yerrs
MLP BI 513
[ountyy
Tea
(012, 912 125,
[annh leat
MIPP ji1
LoGNN Do [512,512] [tleatite]
panh]
LeCCX D (ull) 512 [danritts)
Linear [4,312 [Edeasiin]
Propepation
Propegation
Decoder
Figure 6: The IsoGCN model used for the anisotropic nonlinear heat equation dataset. The numbers in each box denote the number of units. Below the unit numbers, the activation function used for each layer is also shown. @ denotes multiplication in the feature direction, C denotes contraction, and @ denotes the addition in the feature direction.
Table Summary of the hyperparameter setting for both the TFN and Srans-Transfommer. For the parameters not written in the table, we used the default setting in the implementation of https: htt.h....o/aaaaaa uchsut se3 rans former public
pubb
# hidden layers
# NL layers in the self-interaction
1
# channels
16
# maximum rank of the hidden layers
2
# nodes in the radial function
32
rank-O tensor input the propagation block. By stacking this propagation block times, approximate the pth order Taylor expansion of the anisotropic nonlinear heat equation.
For GCN and its variants, simply replaced the IsoGCN layers with the corresponding stacked m (= 2, 5) layers for GCN, GIN, GCNII, and Cluster-GCN, We used an m hop adjacency matrix for SGCN.
We
For the TFN and SE(3)-Transforrs that would fit the GPU because the TFN and Trans-Trrnsfrmmer with almost the same number of parameters as in IsoGCN did not fit on the GPU we used (NVIDIA Tesla V100 with 32 GiB memory). The settings of the hyperparameters are shown in Table
we set the hyperparameters to as many parameters as possible
F.4 RESULT DETAILS
Table 9 shows detailed comparison of the training results. The inclusion of æ in the input features of the baseline models did not improve the performance. addition, if x is included in the input features, loss of the generalization capacity for larger shapes compared the training dataset may result as it extrapolates. The proposed model achieved the best performance compared to the baseline models considered. Therefore, concluded that the essential features regarding the mesh shapes are included in D. Besides, IsoGCN can scale up to meshes with IM vertices as shown in Figure
22

---

Published as a conference paper at ICLR 2021
Table 9; Summary of the test losses (mean squared emorr + the standard error of the mean in the original scale) of the anisotropic nonlinear heat dataset. Here, "x" "Yes", æ is also in the input feature. OOM denotes the cut-of-memory on the applied GPU (32 GiB).
Loss x10 -3
Method
# hops
2
No
16.921 + 0.040
2
Yes
18.483 + 0.025
GIN
5
No
22.961 + 0.056
5
Yes
17.637 + 0.046
2
No
10.427 + 0.028
2
Yes
11.610 + 0.032
GCN
5
No
12.139 + 0.031
5
Yes
11.404 + 0.032
2
No
9.595 = 0.026
2
Yes
9.789 0.028
GCNII
5
No
8.377 = 0.024
5
Yes
9.172 0.028
2
No
7.266 = 0.021
2
Yes
8.532 = 0.023
Cluster-GCh GCN
5
No
8.680 0.024
5
Yes
10.712 + 0.030
2
No
7.317 = 0.021
2
Yes
9.083 + 0.026
SGCN
5
No
6.426 + 0.018
5
Yes
6.519 + 0.020
2
No
15.661 + 0.019
TFN
5
No
OOM
2
14.164
0.018 00M
SE(3)-Trans.
No
2
No
4.674 = 0.014
IsoGCN (Ours)
5
No
2.470 0.008
23

---

Published as a conference paper at ICLR 2021
Ground Truth (FEA)
IsoGCN (Ours)
1.0e+CO
TEMPERATURE
S Training samples
Figure 7: Comparison between (left) samples in the training dataset, (center) ground truth computed through FEA, and (right) IsoGCN inference result. For both the ground truth and inference result, [V] = 1, 011, 301. One can see that IsoGCN can predict the temperature field for a mesh, which much larger than these in the training dataset.
24


Reviews:
[{"review_id": "cVX5lVY6Z7C", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Comments:\nPros:\n1.\tThis paper gives a comprehensive derivation of propositions used in the construction of isoGCN. \n2.\tThe properties provided are clear\n3.\tThe idea to realize the purpose of being isometric transformation invariant and equivariant is good.\n\nCons:\n1.\tMore explanations about the superiority of isometric transformation invariance and equivariance need to be addressed since this purpose seems quite important throughout this paper. Besides, how this purpose helps to achieve the superiority needs to be answered.\n2.\tHow to interpret the adjacency matrix up to m hops, what is the construction of the adjacency matrix mentioned in the paper?\n3.\tIn the experiment results, though the proposed isoGCN gave the minimum loss when comparing with other GCN variants, the scale of loss is around 10^(-5/-6) which may not give a significant goodness.\n4.\tIn table 4, though isoGCN gave the least running time, it also gave larger loss compared with the method FrontISTR. How to explain the trade-off between running time and the loss?\n5.     The paper is too theoretical and does not fit to ICLR.  Real impact and real application should be emphasized more.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "8U1uaT3XD6V", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper proposes a network that operates on features of graphs that are embedded in a d-dim Euclidean space. The paper considers equivariance to a group G that is the direct product of permutations of N points and Euclidean transformations. The features they consider are tensor products of the N-dimensional natural representations of permutations and the d-dimensional standard representation of O(d). From the coordinates, an “isometric adjacency matrix” is created, which is such a tensor. This matrix is combined in various G-equivariant ways with the features and then linearly combined with learnable weights to create new features. These operations are interleaved with non-linearities to form the network. The authors compare to several graph network methods and show competitive performance on several tasks.\n\nStrengths:\n-\tThe authors build a bridge between the global permutation equivariant methods on graphs, as they consider matrix-like features and equivariant maps between them, and Euclidean-equivariant methods on point clouds. This combination seems a powerful approach to point-cloud neural networks.\n\nWeaknesses:\n-\tImportant pieces of prior work are missing from the related work section. The paper seems to be strongly related to Tensor Field Networks (TFN) (Thomas et al. 2018), as both define Euclidean and permutation equivariant convolutions on point clouds / graphs. Furthermore, there are several other methods that operate on graph that are embedded in a Euclidean space, such as SchNet (Schütt et al 2017). The graph network methods currently discussed all do not include the point coordinates in their operations. Lastly, the proposed method operates globally linearly on features on a graph, equivariantly to permutations, which is done in prior work, e.g. Maron 2018. \n-\tThe experimental section only compares to methods that in their convolution are unaware of the point coordinates (except for in the input features). A comparison to coordinate-aware methods, such as TFN or SchNet seems appropriate.\n-\tThe core object, the isometric adjacency matrix G, is ill-defined. In Eq 1 it is defined trough the embedding coordinates and “the transformation invariant rank-2 tensor” T. This object is not defined in the paper, which makes section 3 very confusing to read. In section 3, it appears like that the defined objects D take the role of object G in the above, so what is the role of eq 1? \n-\tIn section 3, the authors speak of “collections of rank-p tensors”. However, these objects seem to actually be tensors of the shape N^a x d^p, where N is the number of nodes, d is the dimensionality of the embedding, and a and p are natural numbers. These objects transform under both permutations and Euclidean transformations in the obvious way. Why not make this fact explicit? That would make section 3 much easier to read. It seems like that when p=0, then a=1, and when p>0, then a=2.  Except for in sec 3.2.2, in which a p=3 tensor has a=1. \n-\tIn Sec 3.2, what are f_in and f_out? Are these the dimensionalities of the tensor product representation? Or do they denote the number of copies of the representation? If it’s the former, I don’t see how the network is equivariant. If it’s the latter, I don’t understand the last paragraph of 3.2.2, which says 1H \\in R^{N x f_in}, which looks like a 0-tensor.\n-\tCan the authors clarify “To achieve translation equivariance, a constant tensor can be added to the output collection of tensors.”? The proposed method seems to only lead to translation invariant features. I do not follow how adding a constant tensor leads to translation equivariance that is not invariance.\n-\tAm I correct in understanding that the method scales cubic with the number of vertices (e.g. eqs 4, 6)? Or is there some sparsity used in the implementation, but not mentioned? Should we expect a method of cubic complexity to scale to 1M vertices? In a naïve implementation, a fast modern GPU with 14.2E12 flops would need 20h for a single 1Mx1M matrix-matrix multiplication (1E18 floating point operations).\n-\tThe authors claim the method scales to 1M vertices, but I cannot find this in the experiments. Table 4 speaks of 155k vertices. How did the authors determine the method scales to 1M vertices? \n\nRecommendation:\nIn its current form, I recommend rejection of this paper. Section 3 is insufficiently clear written, the related work lack important references to prior work and the experiments lack a comparison to potentially strong other methods. This is a shame, because I’d like to see this paper succeed, as the core idea is very strong. Significant improvements in the above criticisms can improve my score.\n\nSuggestions for improvement:\n-\tBe clear about what the G object is and what eq 1 means.\n-\tBe explicit about types the objects, be more explicit about the indices that refer to the permutation representation, to the indices that refer to the Euclidean representation and the indices that refer to copies of the same representation. I think there is an opportunity to be more clear, more explicit, while reducing notational clutter.\n-\tExpand the related work section\n-\tCompare to the strong baselines that use the coordinates.\n-\tProvide argumentation for the claim to scale to 1M vertices.\n\nMinor points:\n-\tEq 7, \\times should be \\otimes?\n-\tEq 14, what is j?\n-\tThe authors write: “A, B and C are X, Y and Z respectively”. Perhaps this could be re-written to the easier to read “A=X, B=Y and C=Z”. This happens each time the word “respectively” is used.\n-\tTable 3 typo, gluster -> cluster\n\n\n### Post rebuttal\nThe authors addressed all my concerns and strongly improved their paper. I think it is now a good candidate for acceptance, as it provides an interesting alternative to / variation on tensor field networks. I raise my rating from 4 to 7.", "extra_scores": {}, "extra_text": {}}, {"review_id": "3DK5_x_eN70", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a graph convolutional network that can be in-/equivariant to isometric transformation. The method is applied to a linear toy problem of predicting the result of several diiferential operators and a nonlinear heat diffusion dataset in comparison to finite element analysis (FEA).\n\nA major problem of the paper is its lack of clarity, which is largely due to the unusual definition of established terms, for example:\n- The contraction (Eq. 4) is usually an operator defined for a single tensor, not two.\n- It is not completely clear to me how Eq. 3 is related to the convolution. Summing over index j look similar to a typical graph convolution, however, here you have an additional index l. Written loosely, a convolution usually has the form $\\sum_j f(i, j) g(j) $, but here you have $\\sum_j f(i, j) g(j, l)$.\n- The notion of a \"collection of tensors\" does not work well here, in particular when aiming to define a convolution (over a space or group). Given the particular application, the notion of tensor fields might be more appropriate.\n\nThe authors mention previous work on equivariant neural networks such as steerable convolutions, tensor field networks, and covariant networks, which they claim are inefficient since they use message-passing, while their approach uses GCNs. This contradicts their statement from above, that GCNs are message-passing neural networks. It should be clarified what exactly makes these approaches less efficient. Moreover, since equivariant neural networks are clearly more suitable for the presented experiments then previous GCNs, some of them should be included in the model comparisons with timings. In particular, going to tensors with large rank (as suggested in Eqs. 8-10) should be rather inefficient compared to equivariant networks using spherical harmonics due to the exponentially increasing number of parameters.\n\nRegarding the timings in Table 4, it is not clear to me, whether the timing include the preprocessing of the adjacency matrix for the inference. Also the caption mentions that a single CPU core was used. Does this refer to both the neural network and the FEA? Otherwise it does not seem like fair comparison.\n\nPros\n====\n- incorporating the various differential operators in the network structures in Sec 3.3 is an interesting idea\n\nCons\n====\n- lacking clarity in method description and notation\n- conceptual differences to steerable convolutions and equivariant networks do not become sufficiently clear\n- experiment section not very strong since comparison to equivariant networks is missing and the timings in Table 4 need some clarification\n\nUpdate: I read the comments of the authors and thank them for the clarifications. The additional baselines improved the paper. I raised my score to reflect that.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper makes an innovative change to the adjacency matrix definition in graph convolutional neural networks (GCNs) (Kipf & Welling, 2017).  The change results in computationally-efficient isometric transformation invariance.   There were a number of concerns raised by reviewers, and the author responses and revisions, and the subsequent discussion, resulted in most of these concerns being satisfactorily addressed.  On reviewer continued to feel the paper was entirely theoretical and therefore not appropriate to ICLR, but that opinion was not shared more broadly and is not held by the area chair.
Decision: Accept (Poster)