EQUIVARIANT GRAPH CONVOLUTIONAL NETWORKS
Year: 2021

Abstract:
Graphs are one of the most important data structures for representing pairwise relations between objects. Specifically, a graph embedded in a Euclidean space is essential to solving real problems, such as physical simulations. A crucial requirement for applying graphs in Euclidean spaces to physical simulations is learning and inferring the isometric transformation invariant and equivariant features in a computationally efficient manner. In this paper, we propose a set of transformation invariant and equivariant models based on graph convolutional networks, called IsoGCNs. We demonstrate that the proposed model has a competitive performance compared to state-of-the-art methods on tasks related to geometrical and physical simulation data. Moreover, the proposed model can scale up to graphs with 1M vertices and conduct an inference faster than a conventional finite element analysis, which the existing equivariant models cannot achieve.

heading: 1 INTRODUCTION
text: Graph-structured data embedded in Euclidean spaces can be utilized in many different fields such as object detection, structural chemistry analysis, and physical simulations. Graph neural networks (GNNs) have been introduced to deal with such data. The crucial properties of GNNs include permutation invariance and equivariance. Besides permutations, isometric transformation invariance and equivariance must be addressed when considering graphs in Euclidean spaces because many properties of objects in the Euclidean space do not change under translation and rotation. Due to such invariance and equivariance, 1) the interpretation of the model is facilitated; 2) the output of the model is stabilized and predictable; and 3) the training is rendered efficient by eliminating the necessity of data augmentation as discussed in the literature (Thomas et al., 2018; Weiler et al.,

heading: 2018; Fuchs et al., 2020).
text: Isometric transformation invariance and equivariance are inevitable, especially when applied to physical simulations, because every physical quantity and physical law is either invariant or equivariant to such a transformation. Another essential requirement for such applications is computational efficiency because the primary objective of learning a physical simulation is to replace a computationally expensive simulation method with a faster machine learning model.
In the present paper, we propose IsoGCNs, a set of simple yet powerful models that provide computationally-efficient isometric transformation invariance and equivariance based on graph convolutional networks (GCNs) (Kipf & Welling, 2017). Specifically, by simply tweaking the definition of an adjacency matrix, the proposed model can realize isometric transformation invariance. Because the proposed approach relies on graphs, it can deal with the complex shapes that are usually presented using mesh or point cloud data structures. Besides, a specific form of the IsoGCN layer can be regarded as a spatial differential operator that is essential for describing physical laws. In addition, we have shown that the proposed approach is computationally efficient in terms of processing graphs
with up to 1M vertices that are often presented in real physical simulations. Moreover, the proposed model exhibited faster inference compared to a conventional finite element analysis approach at the same level of accuracy. Therefore, an IsoGCN can suitably replace physical simulations regarding its power to express physical laws and faster, scalable computation. The corresponding implementation and the dataset are available online1.
The main contributions of the present paper can be summarized as follows:
• We construct isometric invariant and equivariant GCNs, called IsoGCNs for the specified input and output tensor ranks.
• We demonstrate that an IsoGCN model enjoys competitive performance against state-ofthe-art baseline models on the considered tasks related to physical simulations.
• We confirm that IsoGCNs are scalable to graphs with 1M vertices and achieve inference considerably faster than conventional finite element analysis.

heading: 2 RELATED WORK
text: Graph neural networks. The concept of a GNN was first proposed by Baskin et al. (1997); Sperduti & Starita (1997) and then improved by (Gori et al., 2005; Scarselli et al., 2008). Although many variants of GNNs have been proposed, these models have been unified under the concept of message passing neural networks (Gilmer et al., 2017). Generally, message passing is computed with nonlinear neural networks, which can incur a tremendous computational cost. In contrast, the GCN developed by Kipf & Welling (2017) is a considerable simplification of a GNN, that uses a linear message passing scheme expressed as
Hout = σ(ÂHinW ), (1)
where Hin (Hout) is an input (output) feature of the lth layer, Â is a renormalized adjacency matrix with self-loops, and W is a trainable weight. A GCN, among the variants of GNNs, is essential to the present study because the proposed model is based on GCNs for computational efficiency.
Invariant and equivariant neural networks. A function f : X → Y is said to be equivariant to a group G when f(g · x) = g · f(x), for all g ∈ G and x ∈ X , assuming that group G acts on both X and Y . In particular, when f(g · x) = f(x), f is said to be invariant to the group G. Group equivariant convolutional neural networks were first proposed by Cohen & Welling (2016) for discrete groups. Subsequent studies have categorized such networks into continuous groups (Cohen et al., 2018), three-dimensional data (Weiler et al., 2018), and general manifolds (Cohen et al., 2019). These methods are based on CNNs; thus, they cannot handle mesh or point cloud data structures as is. Specifically, 3D steerable CNNs (Weiler et al., 2018) uses voxels (regular grids), which though relatively easy to handle, are not efficient because they represent both occupied and non-occupied parts of an object (Ahmed et al., 2018). In addition, a voxelized object tends to lose the smoothness of its shape, which can lead to drastically different behavior in a physical simulation, as typically observed in structural analysis and computational fluid dynamics.
Thomas et al. (2018); Kondor (2018) discussed how to provide rotation equivariance to point clouds. Specifically, the tensor field network (TFN) (Thomas et al., 2018) is a point cloud based rotation and translation equivariant neural network the layer of which can be written as
H̃ (l)
out,i = w ll H̃
(l) in,i + ∑ k≥0 ∑ j 6=i W lk(xj − xi) H̃ (k) in,j , (2)
W lk(x) = k+l∑ J=|k−l| φlkJ (‖x‖) J∑ m=−J YJm(x/‖x‖)QlkJm, (3)
where H̃ (l) in,i (H̃ (l) out,i) is a type-l input (output) feature at the ith vertex, φ lk J : R≥0 → R is a trainable function, YJm is the mth component of the J th spherical harmonics, and QlkJm is the ClebschCordan coefficient. The SE(3)-Transformer (Fuchs et al., 2020) is a variant of the TFN with selfattention. These models achieve high expressibility based on spherical harmonics and message passing with nonlinear neural networks. However, for this reason, considerable computational resources
1https://github.com/yellowshippo/isogcn-iclr2021
are required. In contrast, the present study allows a significant reduction in the computational costs because it eliminates spherical harmonics and nonlinear message passing. From this perspective, IsoGCNs are also regarded as a simplification of the TFN, as seen in equation 14.
Physical simulations using GNNs. Several related studies, including those by Sanchez-Gonzalez et al. (2018; 2019); Alet et al. (2019); Chang & Cheng (2020) focused on applying GNNs to learn physical simulations. These approaches allowed the physical information to be introduced to GNNs; however, addressing isometric transformation equivariance was out of the scope of their research.
In the present study, we incorporate isometric transformation invariance and equivariance into GCNs, thereby, ensuring the stability of the training and inference under isometric transformation. Moreover, the proposed approach is efficient in processing large graphs with up to 1M vertices that have a sufficient number of degrees of freedom to express complex shapes.

heading: 3 ISOMETRIC TRANSFORMATION INVARIANT AND EQUIVARIANT GRAPH CONVOLUTIONAL LAYERS
text: In this section, we discuss how to construct IsoGCN layers that correspond to the isometric invariant and equivariant GCN layers. To formulate a model, we assume that: 1) only attributes associated with vertices and not edges; and 2) graphs do not contain self-loops. Here, G = (V, E) denotes a graph and d denotes the dimension of a Euclidean space. In this paper, we refer to tensor as geometric tensors, and we consider a (discrete) rank-p tensor field H(p) ∈ R|V|×f×dp , where |V| denotes the number of vertices and f ∈ Z+ (Z+ denotes the positive integers). Here, f denotes the number of features (channels) of H(p), as shown in Figure 1 (a). With the indices, we denote H(p)i;g;k1k2...kp , where i permutes under the permutation of vertices and k1, . . . , kp refers to the Euclidean representation. Thus, under the permutation, π : H(p)i;g;k1k2...kp 7→ H (p) π(i);g;k1k2...kp , and under orthogonal
transformation, U : H(p)i;g;k1k2...kp 7→ ∑ l1,l2,...,lp Uk1l1Uk2l2 . . . UkplpH (p) i;g;l1l2...lp .

heading: 3.1 CONSTRUCTION OF AN ISOMETRIC ADJACENCY MATRIX
text: Before constructing an IsoGCN, an isometric adjacency matrix (IsoAM), which is at the core of the IsoGCN concept must be defined. The proof of each proposition can be found in Appendix B.
An IsoAM G ∈ R|V|×|V|×d is defined as: Rd 3 Gij;;: := ∑ k,l∈V,k 6=l Tijkl(xk − xl), (4)
where Gij;;: is a slice in the spatial index of G, xi ∈ Rd is the position of the ith vertex (rank1 tensor), and Tijkl ∈ Rd×d is an untrainable transformation invariant and orthogonal transformation equivariant rank-2 tensor. Note that we denote Gij;;k to be consistent with the no-
tation of H(p)i;g;k1k2...kp because i and j permutes under the vertex permutation and k represents the spatial index while the number of features is always 1. The IsoAM can be viewed as a weighted adjacency matrix for each direction and reflects spatial information while the usual weighted adjacency matrix cannot because a graph has only one adjacency matrix. If the size of the set {Gij;;: 6= 0}j is greater than or equal to d, then it can be deemed to be a frame, which is a generalization of a basis. For the simplest case, one can define Tijkl = δilδjkAijI (Figure 1 (b)), where δij is the Kronecker delta, A is the adjacency matrix of the graph, and I is the identity matrix that is the simplest rank-2 tensor. In another case, Tijkl can be determined from the geometry of a graph, as defined in equation 16. Nevertheless, in the bulk of this section, we retain Tijkl abstract to cover various forms of interaction, such as position-aware GNNs (You et al., 2019). Here, G is composed of only untrainable parameters and thus can be determined before training.
Proposition 3.1. IsoAM defined in equation 4 is both translation invariant and orthogonal transformation equivariant, i.e., for any isometric transformation ∀t ∈ R3,U ∈ O(d), T : x 7→ Ux+ t,
T : Gij;;k 7→ ∑ l UklGij;;l. (5)
Based on the definition of the GCN layer in the equation 1, let G ∗ H(0) ∈ R|V|×f×d denote the convolution between G and the rank-0 tensor field H(0) ∈ R|V|×f (f ∈ Z+) as follows:
(G ∗ H(0))i;g;k := ∑ j Gij;;kH (0) j;g;. (6)
With a rank-1 tensor field H(1) ∈ R|V|×f×d, let G H(1) ∈ R|V|×f and G G ∈ R|V|×|V| denote the contractions which are defined as follows:
(G H(1))i;g; := ∑ j,k Gij;;kH (1) j;g;k, (G G)il;; := ∑ j,k Gij;;kGjl;k. (7)
The contraction of IsoAMs G G can be interpreted as the inner product of each component in the IsoAMs. Thus, the subsequent proposition follows. Proposition 3.2. The contraction of IsoAMs G G is isometric transformation invariant, i.e., for any isometric transformation ∀t ∈ R3,U ∈ O(d), T : x 7→ Ux + t, G G 7→ G G.
With a rank-p tensor field H(p) ∈ R|V|×f×dp , let G⊗H(p) ∈ R|V|×f×d1+p . and G⊗G ∈ R|V|×|V|×d2 denote the tensor products defined as follows:
(G⊗ H(p))i;g;km1m2...mp := ∑ j Gij;;kH (p) j;g;m1m2...mp , (8)
(G⊗G)il;;k1k2 := ∑ j Gij;;k1Gjl;;k2 . (9)
The tensor product of IsoAMs G⊗G can be interpreted as the tensor product of each of the IsoAMs components. Thus, the subsequent proposition follows: Proposition 3.3. The tensor product of the IsoAMs G⊗G is isometric transformation equivariant in terms of the rank-2 tensor, i.e., for any isometric transformation ∀t ∈ R3,U ∈ O(d), T : x 7→ Ux + t, and ∀i, j ∈ 1, . . . , |V|, (G⊗G)ij;;k1k2 7→ Uk1l1Uk2l2(G⊗G)ij;;l1l2 .
This proposition is easily generalized to the tensors of higher ranks by defining the pth tensor power of G as follows: ⊗0 G = 1, ⊗1 G = G, and ⊗p G = ⊗p−1 G⊗G. Namely, ⊗p G is isometric transformation equivariant in terms of rank-p tensor. Therefore, one can see that (
⊗p G)⊗ H(q) = ( ⊗p−1 G) ⊗ (G ⊗ H(q)). Moreover, the convolution can be generalized for ⊗p G and the rank-0 tensor field H(0) ∈ R|V|×f as follows:[( p⊗
G ) ∗ H(0) ] i;g;k1k2...kp = ∑ j ( p⊗ G ) ij;;k1k2...kp H (0) j;g;. (10)

heading: The contraction can be generalized for
text: ⊗p G and the rank-q tensor field H(q) ∈ R|V|×f×dq (p ≥ q) as specified below:[( p⊗
G ) H(q) ] i;g;k1k2...kp−q = ∑ j,m1,m2,...,mq ( p⊗ G ) ij;;k1k2...kp−qm1m2...mq H(q)j;g;m1m2...mq .
(11) For the case p < q, the contraction can be defined similarly.

heading: 3.2 CONSTRUCTION OF ISOGCN
text: Using the operations defined above, we can construct IsoGCN layers, which take the tensor field of any rank as input, and output the tensor field of any rank, which can differ from those of the input.

heading: In addition, one can show that these layers are also equivariant under the vertex permutation, as
text: discussed in Maron et al. (2018).

heading: 3.2.1 ISOMETRIC TRANSFORMATION INVARIANT LAYER

heading: As can be seen in Proposition 3.1, the contraction of IsoAMs is isometric transformation invariant. Therefore, for the isometric transformation invariant layer with a rank-0 input tensor field
text: f : R|V|×fin 3 H(0)in 7→ H (0) out ∈ R|V|×fout (fin, fout ∈ Z+), the activation function σ, and the
trainable parameter matrix W ∈ Rfin×fout can be constructed as H(0)out = σ ( (G G)H(0)in W ) .
By defining L := G G ∈ R|V|×|V|, it can be simplified as H(0)out = σ ( LH (0) in W ) , which has the
same form as a GCN (equation 1), with the exception that Â is replaced with L.
An isometric transformation invariant layer with the rank-p input tensor field H(p)in ∈ R|V|×fin×d p can be formulated as H(0)out = Fp→0(H (p) in ) = σ ([⊗p G H(p)in ]W) . If p = 1, such approaches utilize the inner products of the vectors in Rd, these operations correspond to the extractions of a relative distance and an angle of each pair of vertices, which are employed in Klicpera et al. (2020).

heading: 3.2.2 ISOMETRIC TRANSFORMATION EQUIVARIANT LAYER
text: To construct an isometric transformation equivariant layer, one can use linear transformation, convolution and tensor product to the input tensors. If both the input and the output tensor ranks are greater than 0, one can apply neither nonlinear activation nor bias addition because these operations will cause an inappropriate distortion of the isometry because isometric transformation does not commute with them in general. However, a conversion that uses only a linear transformation, convolution, and tensor product does not have nonlinearity, which limits the predictive performance of the model. To add nonlinearity to such a conversion, we can first convert the input tensors to rank-0 ones, apply nonlinear activations, and then multiply them to the higher rank tensors.
The nonlinear isometric transformation equivariant layer with the rank-m input tensor field H(m)in ∈ R|V|×fin×dm and the rank-l (m ≤ l) output tensor H(l)out ∈ R|V|×fout×d l can be defined as:
H(l)out = Fm→0 ( H(m)in ) × Fm→l ( H(m)in ) , Fm→l ( H(m)in ) =
[ l−m⊗
G ] ⊗ H(m)in Wml, (12)
where × denotes multiplication with broadcasting and Wml ∈ Rfin×fout are trainable weight matrices multiplied in the feature direction. If m = 0, we regard G⊗ H(0) as G ∗ H(0). If m = l, one can add the residual connection (He et al., 2016) in equation 12. If m > l,
H(l)out = Fm→0 ( H(m)in ) × Fm→l ( H(m)in ) , Fm→l ( H(m)in ) =
[ m−l⊗
G ] H(m)in Wml. (13)
In general, the nonlinear isometric transformation equivariant layer with the rank-0 to rank-M input tensor field {H(m)in }Mm=0 and the rank-l output tensor field H (l) out can be defined as:
H(l)out = H (l) in W + M∑ m=0 fgather ({ Fk→0(H (k) in ) }M k=0 ) × Fm→l ( H(m)in ) , (14)
where fgather denotes a function such as summation, product and concatenation in the feature direction. One can see that this layer is similar to that in the TFN (equation 2), while there are no spherical harmonics and trainable message passing.

heading: To be exact, the output of the layer defined above is translation invariant. To output translation equivariant variables such as the vertex positions after deformation (which change accordingly with
text: the translation of the input graph), one can first define the reference vertex position xref for each graph, then compute the translation invariant output using equation 14, and finally, add xref to the output. For more detailed information on IsoGCN modeling, see Appendix D.

heading: 3.3 EXAMPLE OF ISOAM
text: The IsoGCN G is defined in a general form for the propositions to work with various classes of graph. In this section, we concretize the concept of the IsoAM to apply an IsoGCN
to mesh structured data. Here, a mesh is regarded as a graph regarding the points in the mesh as vertices of the graph and assuming two vertices are connected when they share the same cell. A concrete instance of IsoAM D̃,D ∈ R|V|×|V|×d is defined as follows:
Table 1: Correspondence between the differential operators and the expressions using the IsoAM D̃.
Differential op. Expression Gradient D̃ ∗ H(0) Divergence D̃ H(1) Laplacian D̃ D̃H(0) Jacobian D̃⊗ H(1) Hessian D̃⊗ D̃ ∗ H(0)
D̃ij;;k = Dij;;k − δij ∑ l Dil;;k, (15)
Dij;;: = M−1i xj − xi ‖xj − xi‖2 wijAij(m), (16)
Mi = ∑ l xl − xi ‖xl − xi‖ ⊗ xl − xi‖xl − xi‖ wilAil(m), (17)
where R|V|×|V| 3 A(m) := min (∑mk=1 Ak, 1) is an adjacency matrix up to m hops and wij ∈ R is an untrainable weight between the ith and jth vertices that is determined depending on the tasks2. By regarding Tijkl = δilδjkM −1 i wijAij(m)/‖xj − xi‖2 in equation 4, one can see that D is qualified as an IsoAM. Because a linear combination of IsoAMs is also an IsoAM, D̃ is an IsoAM. Thus, they provide both translation invariance and orthogonal transformation equivariance. D̃ can be obtained only from the mesh geometry information, thus can be computed in the preprocessing step.
Here, D̃ is designed such that it corresponds to the gradient operator model used in physical simulations (Tamai & Koshizuka, 2014; Swartz & Wendroff, 1969). As presented in Table 1 and Appendix C, D̃ is closely related to many differential operators, such as the gradient, divergence, Laplacian, Jacobian, and Hessian. Therefore, the considered IsoAM plays an essential role in constructing neural network models that are capable of learning differential equations.

heading: 4 EXPERIMENT
text: To test the applicability of the proposed model, we composed the following two datasets: 1) a differential operator dataset of grid meshes; and 2) an anisotropic nonlinear heat equation dataset of meshes generated from CAD data. In this section, we discuss our machine learning model, the definition of the problem, and the results for each dataset.
Using D̃ defined in Section 3.3, we constructed a neural network model considering an encodeprocess-decode configuration (Battaglia et al., 2018). The encoder and decoder were comprised of component-wise MLPs and tensor operations. For each task, we tested m = 2, 5 in equation 16 to investigate the effect of the number of hops considered. In addition to the GCN (Kipf & Welling, 2017), we chose GIN (Xu et al., 2018), SGCN (Wu et al., 2019), Cluster-GCN (Chiang et al., 2019), and GCNII (Chen et al., 2020) as GCN variant baseline models. For the equivariant models, we chose the TFN (Thomas et al., 2018) and SE(3)-Transformer (Fuchs et al., 2020) as the baseline. We implemented these models using PyTorch 1.6.0 (Paszke et al., 2019) and PyTorch Geometric 1.6.1 (Fey & Lenssen, 2019). For both the TFN and SE(3)-Transformer, we used implementation of Fuchs et al. (2020) 3 because the computation of the TFN is considerably faster than the original implementation, as claimed in Fuchs et al. (2020). For each experiment, we minimized the mean squared loss using the Adam optimizer (Kingma & Ba, 2014). The corresponding implementation and the dataset will be made available online. The details of the experiments can be found in Appendix E and F.

heading: 4.1 DIFFERENTIAL OPERATOR DATASET
text: To demonstrate the expressive power of IsoGCNs, we created a dataset to learn the differential operators. We first generated a pseudo-2D grid mesh randomly with only one cell in the Z direction and 10 to 100 cells in the X and Y directions. We then generated scalar fields on the grid meshes
2Mi is invertible when the number of independent vectors in {xl − xi}l is greater than or equal to the space dimension d, which is true for common meshes, e.g., a solid mesh in 3D Euclidean space.
3https://github.com/FabianFuchsML/se3-transformer-public
and analytically calculated the gradient, Laplacian, and Hessian fields. We generated 100 samples for each train, validation, and test dataset. For simplicity, we set wij = 1 in equation 16 for all (i, j) ∈ E . To compare the performance with the GCN models, we simply replaced an IsoGCN layer with a GCN or its variant layers while keeping the number of hops m the same to enable a fair comparison. We adjusted the hyperparameters for the equivariant models to ensure that the number of parameters in each was almost the same as that in the IsoGCN model. For more details regarding the model architecture, see Appendix E. We conducted the experiments using the following settings: 1) inputting the scalar field and predicting the gradient field (rank-0→ rank-1 tensor); 2) inputting the scalar field and predicting the Hessian field (rank-0→ rank-2 tensor); 3) inputting the gradient field and predicting the Laplacian field (rank-1→ rank-0 tensor); and 4) inputting the gradient field and predicting the Hessian field (rank-1→ rank-2 tensor). Figure 2 and Table 2 present a visualization and comparison of predictive performance, respectively. The results show that an IsoGCN outperforms other GCN models for all settings. This is because the IsoGCN model has information on the relative position of the adjacency vertices, and thus understands the direction of the gradient, whereas the other GCN models cannot distinguish where the adjacencies are, making it nearly impossible to predict the gradient directions. Adding the vertex positions to the input feature to other GCN models exhibited a performance improvement, however as the vertex position is not a translation invariant feature, it could degrade the predictive performance of the models. Thus, we did not input x as a vertex feature to the IsoGCN model or other equivariant models to retain their isometric transformation invariant and equivariant natures. IsoGCNs perform competitively against other equivariant models with shorter inference time as shown in Ta-
ble 7. As mentioned in Section 3.3, D̃ corresponds to the gradient operator, which is now confirmed in practice.

heading: 4.2 ANISOTROPIC NONLINEAR HEAT EQUATION DATASET
text: To apply the proposed model to a real problem, we adopted the anisotropic nonlinear heat equation. We considered the task of predicting the time evolution of the temperature field based on the initial temperature field, material property, and mesh geometry information as inputs. We randomly selected 82 CAD shapes from the first 200 shapes of the ABC dataset (Koch et al., 2019), generate first-order tetrahedral meshes using a mesh generator program, Gmsh (Geuzaine & Remacle, 2009), randomly set the initial temperature and anisotropic thermal conductivity, and finally conducted a finite element analysis (FEA) using the FEA program FrontISTR4 (Morita et al., 2016; Ihara et al., 2017).
For this task, we set wij = V effectivej /V effective i , where V effective i denotes the effective volume of the ith vertex (equation 46.) Similarly to the differential operator dataset, we tested the number of hops m = 2, 5. However because we put four IsoAM operations in one model, the number of hops visible from the model is 8 (m = 2) or 20 (m = 5). As is the case with the differential operator dataset, we replaced an IsoGCN layer accordingly for GCN or its variant models. In the case of k = 2, we reduced the number of parameters for each of the equivariant models to fewer than the IsoGCN model because they exceeded the memory of the GPU (NVIDIA Tesla V100 with 32 GiB memory) with the same number of parameters. In the case of k = 5, neither the TFN nor the SE(3)Transformer fits into the memory of the GPU even with the number of parameters equal to 10. For more details about the dataset and the model, see Appendix F.
Figure 3 and Table 3 present the results of the qualitative and quantitative comparisons for the test dataset. The IsoGCN demonstrably outperforms all other baseline models. Moreover, owing to the computationally efficient isometric transformation invariant nature of IsoGCNs, it also achieved a high prediction performance for the meshes that had a significantly larger graph than those considered in the training dataset. The IsoGCN can scale up to 1M vertices, which is practical and is considerably greater than that reported in Sanchez-Gonzalez et al. (2020). Therefore, we conclude that IsoGCN models can be trained on relatively smaller meshes5 to save the training time and then used to apply the inference to larger meshes without observing significant performance deterioration.
Table 4 reports the preprocessing and inference computation time using the equivariant models with m = 2 as the number of hops and FEA using FrontISTR 5.0.0. We varied the time step (∆t =
4https://github.com/FrontISTR/FrontISTR. We applied a private update to FrontISTR to deal with the anisotropic heat problem, which will be also made available online.
5However, it should also be sufficiently large to express sample shapes and fields.
1.0, 0.5) for the FEA computation to compute the t = 1.0 time evolution thus, resulting in different computation times and errors compared to an FEA with ∆t = 0.01, which was considered as the ground truth. Clearly, the IsoGCN is 3- to 5- times faster than the FEA with the same level of accuracy, while other equivariant models have almost the same speed as FrontISTR with ∆t = 0.5.
5 CONCLUSION
In this study, we proposed the GCNbased isometric transformation invariant and equivariant models called IsoGCN. We discussed an example of an isometric adjacency matrix (IsoAM) that was closely related to the essential differential operators. The experiment results confirmed that the proposed model leveraged the spatial structures and can deal with large-scale graphs. The computation time of the IsoGCN model is significantly shorter than the FEA, which other equivariant models cannot achieve. Therefore, IsoGCN must be the first choice to learn physical simulations because of its computational efficiency as well as isometric transformation invariance and equivariance. Our demonstrations were conducted on the mesh structured dataset based on the FEA results. However, we expect IsoGCNs to be applied to various domains,
such as object detection, molecular property prediction, and physical simulations using particles.

heading: ACKNOWLEDGMENTS
text: The authors gratefully acknowledge Takanori Maehara for his helpful advice and NVIDIA for hardware donations. This work was supported by JSPS KAKENHI Grant Number 19H01098.

heading: Eman Ahmed, Alexandre Saint, Abd El Rahman Shabayek, Kseniya Cherenkova, Rig Das, Gleb
text: Gusev, Djamila Aouada, and Bjorn Ottersten. A survey on deep learning advances on different 3d data representations. arXiv preprint arXiv:1808.01462, 2018.
Ferran Alet, Adarsh Keshav Jeewajee, Maria Bauza Villalonga, Alberto Rodriguez, Tomas LozanoPerez, and Leslie Kaelbling. Graph element networks: adaptive, structured computation and memory. In ICML, 2019.
Igor I Baskin, Vladimir A Palyulin, and Nikolai S Zefirov. A neural device for searching direct correlations between structures and properties of chemical compounds. Journal of chemical information and computer sciences, 37(4):715–721, 1997.

heading: Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi,
text: Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01261, 2018.
Kai-Hung Chang and Chin-Yi Cheng. Learning to simulate and design for structural engineering. arXiv preprint arXiv:2003.09103, 2020.
Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. arXiv preprint arXiv:2007.02133, 2020.

heading: Wei-Lin Chiang, Xuanqing Liu, Si Si, Yang Li, Samy Bengio, and Cho-Jui Hsieh. Cluster-gcn: An
text: efficient algorithm for training deep and large graph convolutional networks. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 257–266, 2019.
Taco Cohen and Max Welling. Group equivariant convolutional networks. In International conference on machine learning, pp. 2990–2999, 2016.
Taco S Cohen, Mario Geiger, Jonas Köhler, and Max Welling. Spherical cnns. In ICLR, 2018.
Taco S Cohen, Maurice Weiler, Berkay Kicanaoglu, and Max Welling. Gauge equivariant convolutional networks and the icosahedral cnn. ICML, 2019.
Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019.
Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling. Se (3)-transformers: 3d rototranslation equivariant attention networks. Advances in Neural Information Processing Systems, 33, 2020.
Christophe Geuzaine and Jean-François Remacle. Gmsh: a three-dimensional finite element mesh generator with built-in pre- and post-processing facilities. International Journal for Numerical Methods in Engineering, 79(11):1309–1331, 2009.

heading: Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural
text: message passing for quantum chemistry. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pp. 1263–1272. JMLR. org, 2017.
Marco Gori, Gabriele Monfardini, and Franco Scarselli. A new model for learning in graph domains. In Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005., volume 2, pp. 729–734. IEEE, 2005.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016.
Yu Ihara, Gaku Hashimoto, and Hiroshi Okuda. Web-based integrated cloud cae platform for largescale finite element analysis. Mechanical Engineering Letters, 3:17–00520, 2017.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In ICLR, 2017.
Johannes Klicpera, Janek Groß, and Stephan Günnemann. Directional message passing for molecular graphs. In ICLR, 2020.
Sebastian Koch, Albert Matveev, Zhongshi Jiang, Francis Williams, Alexey Artemov, Evgeny Burnaev, Marc Alexa, Denis Zorin, and Daniele Panozzo. Abc: A big cad model dataset for geometric deep learning. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Risi Kondor. N-body networks: a covariant hierarchical neural network architecture for learning atomic potentials. arXiv preprint arXiv:1803.01588, 2018.
Haggai Maron, Heli Ben-Hamu, Nadav Shamir, and Yaron Lipman. Invariant and equivariant graph networks. arXiv preprint arXiv:1812.09902, 2018.
Naoki Morita, Kazuo Yonekura, Ichiro Yasuzumi, Mitsuyoshi Tsunori, Gaku Hashimoto, and Hiroshi Okuda. Development of 3× 3 dof blocking structural elements to enhance the computational intensity of iterative linear solver. Mechanical Engineering Letters, 2:16–00082, 2016.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10), pp. 807–814, 2010.

heading: Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor
text: Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024–8035. Curran Associates, Inc., 2019.

heading: Alvaro Sanchez-Gonzalez, Nicolas Heess, Jost Tobias Springenberg, Josh Merel, Martin Riedmiller,
text: Raia Hadsell, and Peter Battaglia. Graph networks as learnable physics engines for inference and control. arXiv preprint arXiv:1806.01242, 2018.

heading: Alvaro Sanchez-Gonzalez, Victor Bapst, Kyle Cranmer, and Peter Battaglia. Hamiltonian graph
text: networks with ode integrators. arXiv preprint arXiv:1909.12790, 2019.
Alvaro Sanchez-Gonzalez, Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter W Battaglia. Learning to simulate complex physics with graph networks. arXiv preprint arXiv:2002.09405, 2020.

heading: Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini.
text: The graph neural network model. IEEE Transactions on Neural Networks, 20(1):61–80, 2008.
Alessandro Sperduti and Antonina Starita. Supervised neural networks for the classification of structures. IEEE Transactions on Neural Networks, 8(3):714–735, 1997.
Blair Swartz and Burton Wendroff. Generalized finite-difference schemes. Mathematics of Computation, 23(105):37–49, 1969.
Tasuku Tamai and Seiichi Koshizuka. Least squares moving particle semi-implicit method. Computational Particle Mechanics, 1(3):277–305, 2014.

heading: Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick
text: Riley. Tensor field networks: Rotation-and translation-equivariant neural networks for 3d point clouds. arXiv preprint arXiv:1802.08219, 2018.

heading: Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco S Cohen. 3d steerable
text: cnns: Learning rotationally equivariant features in volumetric data. In NeurIPS, pp. 10381–10392, 2018.
Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. Simplifying graph convolutional networks. In ICML, pp. 6861–6871. PMLR, 2019.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint arXiv:1810.00826, 2018.
Jiaxuan You, Rex Ying, and Jure Leskovec. Position-aware graph neural networks. arXiv preprint arXiv:1906.04817, 2019.

heading: A NOTATION
text: G A graph V A vertex set |V| The number of vertices E An edge set Z+ The positive integers d The dimension of the Euclidean space
xi The position of the ith vertex
xik Element k of xi G ∈ R|V|×|V|×d The isometric adjacency matrix (IsoAM) (equation 4) Gij;;: ∈ Rd Slice of G in the spatial index (equation 4) Gij;;k ∈ R Element (i, j, k) of G H(p) ∈ R|V|×f×dp A rank-p tensor field tensor (f, p ∈ Z+) H(p)i;g;k1k2...kp Element (i; g; k1, k2, . . . , kp) of H
(p). i refers to the permutation representation, k1, . . . kp refer to the Euclidean representation, and g denotes the feature index (See section 3).[
p⊗ G ] ∗ H(0) Convolution of the pth power of G and rank-0 tensor field H(0) (equa-
tion 6, equation 10)[ p⊗
G ] H(q) Contraction of the pth power of G and rank-q tensor fields (equation 7,
equation 11)[ p⊗
G ] ⊗ H(q) Tensor product of the pth power of G and rank-q tensor fields H(q)
(equation 8)
H(p)in The rank-p input tensor field of the considered layer H(p)out The rank-p output tensor field of the considered layer
σ The activation function
W The trainable parameter matrix
A ∈ R|V|×|V| An adjacency matrix δij The Kronecker delta
V effectivei The effective volume of the ith vertex (equation 46)
V meani The mean volume of the ith vertex (equation 47) D̃ ∈ R|V|×|V| A concrete instance of IsoAM (equation 15)

heading: B PROOFS OF PROPOSITIONS
text: In this section, we present the proofs of the propositions described in Section 3. Let R3 3 g(xl,xk) = (xk − xl). Note that G is expressed using g(xi,xj) as Gij;;: =∑ k,l∈V,k 6=l Tijklg(xl,xk).

heading: B.1 PROOF OF PROPOSITION 3.1
text: Proof. First, we demonstrate the invariance with respect to the translation with ∀t ∈ Rd. g(xi,xj) is transformed invariantly as follows under translation:
g(xi + t,xj + t) = [xj + t− (xi + t)] = (xj − xi) = g(xi,xj). (18)
By definition, Tijkl is also translation invariant. Thus,∑ k,l∈V,k 6=l Tijklg(xl + t,xk + t) = ∑ k,l∈V,k 6=l Tijklg(xl,xk)
= Gij;;:. (19)
We then show an equivariance regarding the orthogonal transformation with ∀U ∈ O(d). g(xi,xj) is transformed as follows by orthogonal transformation:
g(Uxi,Uxj) = Uxj −Uxi = U(xj − xi) = Ug(xi,xj). (20)
By definition, Tijkl is transformed to UTijklU−1 by orthogonal transformation. Thus,∑ k,l∈V,k 6=l UTijklU −1g(Uxl,Uxk) = ∑ k,l∈V,k 6=l UTijklU −1Ug(xl,xk)
= UGij;;:. (21)
Therefore, G is both translation invariant and an orthogonal transformation equivariant.

heading: B.2 PROOF OF PROPOSITION 3.2
text: Proof. Here, G G is translation invariant because G is translation invariant. We prove rotation invariance under an orthogonal transformation ∀U ∈ O(n). In addition, G G is transformed under U as follows:∑
j,k
Gij;;kGjl;;k 7→ ∑
j,k,m,n
UkmGij;;mUknGjl;;n
= ∑
j,k,m,n
UkmUknGij;;mGjl;;n
= ∑
j,k,m,n
UTmkUknGij;;mGjl;;n
= ∑ j,m,n δmnGij;;mGjl;;n (∵ property of the orthogonal matrix)
= ∑ j Gij;;mGjl;;m
= ∑ j,k Gij;;kGjl;;k. (∵ Change the dummy index m→ k) (22)
Therefore, G G is isometric transformation invariant.

heading: B.3 PROOF OF PROPOSITION 3.3
text: Proof. G⊗G is transformed under ∀U ∈ O(n) as follows:
∑ j Gij;;kGjl;;m 7→ ∑ n,o UknGij;;nUmoGjl;;o
= ∑ n,o UknGij;;nGjl;;oUTom. (23)
By regarding Gij;;nGjl;;o as one matrix Hno, it follows the coordinate transformation of rank-2 tensor UHUT for each i, j, and l.

heading: C PHYSICAL INTUITION OF D̃
text: In this section, we discuss the connection between the concrete IsoAM example D̃ and the differential operators such as the gradient, divergence, the Laplacian, the Jacobian, and the Hessian operators.
Let φi ∈ R denote a rank-0 tensor (scalar) at the ith vertex. Let us assume a partial derivative model of a rank-0 tensor φ at the ith vertex regarding the kth axis (∂φ/∂xk)i ∈ R (k ∈ {1, . . . , d}), that is based on the gradient model in the least squares moving particle semi-implicit method (Tamai & Koshizuka, 2014).
( ∂φ
∂xk ) i :=M−1i ∑ j φj − φi ‖xj − xi‖ xjk − xik ‖xj − xi‖ wijAij(m) (24)
= ∑ j Dijk(φj − φi), (25)
Mi = ∑ l xl − xi ‖xl − xi‖ ⊗ xl − xi‖xl − xi‖ wilAil(m). (26)
Although one could define wij as a function of the distance ‖xj − xi‖, wij was kept constant with respect to the distance required to maintain the simplicity of the model with fewer hyperparameters.

heading: C.1 GRADIENT
text: D̃ can be viewed as a Laplacian matrix based on D; however, D̃ ∗ H(0) can be interpreted as the gradient within the Euclidean space. Let ∇ H(0) ∈ R|V|×f×d be an approximation of the gradient of H(0). Using equation 25, the gradient model can be expressed as follows:
( ∇ H(0) ) i;g;k = ∂H(0)i;g; ∂xk
(27)
= Dijk(H (0) j;g; − H (0) i;g;). (28)
Using this gradient model, we can confirm that (D̃ ∗ H(0))i;g;k = (∇ H(0))i;glk because
( D̃ ∗ H(0) ) i;g;k = ∑ j D̃ij;;kH (0) j;g; (29)
= ∑ j (Dij;;k − δij ∑ l Dil;;k)H (0) j;g;
= ∑ j Dij;;kH (0) j;g; − ∑ j,l δijDil;;kH (0) j;g;
= ∑ j Dij;;kH (0) j;g; − ∑ l Dil;;kH (0) i;g;
= ∑ j Dij;;kH (0) j;g; − ∑ j Dij;;kH (0) i;g; (∵ Change the dummy index l→ j)
= ∑ j Dij;;k(H (0) j;g; − H (0) i;g;)
= ( ∇ H(0) ) i;g;k . (30)
Therefore, D̃∗ can be interpreted as the gradient operator within a Euclidean space.

heading: C.2 DIVERGENCE
text: We show that D̃ H(1) corresponds to the divergence. Using D, the divergence model ∇ · H(1) ∈ R|V|×f is expressed as follows:
( ∇ · H(1) ) i;g; = (∑ k ∂ H(1) ∂xk ) i;g;
(31)
= ∑ j,k Dij;;k(H (1) j;g;k − H (1) i;g;k). (32)
Then, D̃ H(1) is
(D̃ H(1))i;g; = ∑ j,k D̃ij;;kH (1) i;g;k
= ∑ j,k
( Dij;;k − δij
∑ l D ) H(1)i;g;k
= ∑ j,k Dij;;kH (1) j;g;k − ∑ l,k Dil;;kH (1) i;g;k
= ∑ j,k Dij;;k(H (1) j;g;k − H (1) i;g;k) (∵ Change the dummy index l→ j)
= (∇ · H(1))i;g;. (33)

heading: C.3 LAPLACIAN OPERATOR
text: We prove that D̃ D̃ corresponds to the Laplacian operator within a Euclidean space.
Using equation 25, the Laplacian model∇2 H(0) ∈ R|V|×f can be expressed as follows:
( ∇2 H(0) ) i;g; := ∑ k [ ∂ ∂xk ( ∂H ∂xk ) i ] i;g;
= ∑ j,k Dij;;k [( ∂H ∂xk ) j;g; − ( ∂H ∂xk ) i;g; ]
= ∑ j,k Dij;;k [∑ l Djl;;k(H (0) l;g; − H (0) j;g;)− ∑ l Dil;;k(H (0) l;g; − H (0) i;g;) ] = ∑ j,k,l Dij;;k(Djl;;k − Dil;;k)(H(0)l;g; − H (0) j;g;). (34)
Then, (D̃ D̃)H(0) is ((D̃ D̃)H(0))i;g; = ∑ j,k,l D̃ij;;kD̃jl;;kH (0) l;g;
= ∑ j,k,l
( Dij;;k − δij
∑ m Dim;;k
)( Djl;;k − δjl
∑ n Djn;;k ) H(0)l;g;
= ∑ j,k,l Dij;;kDjl;;kH (0) l;g; − ∑ j,k,n Dij;;kDjn;;kH (0) j;g;
− ∑ k,l,m Dim;;kDil;;kH (0) l;g; + ∑ k,m,n Dim;;kDin;;kH (0) i;g;
= ∑ j,k,l Dij;;kDjl;;kH (0) l;g; − ∑ j,k,n Dij;;kDjn;;kH (0) j;g;
− ∑ k,l,j Dij;;kDil;;kH (0) l;g; + ∑ k,j,n Dij;;kDin;;kH (0) i;g;
(∵ Change the dummy index m→ j for the third and fourth terms)
= ∑ j,k,l Dij;;k(Djl;;k − Dil;;k)(H(0)l;g; − H (0) j;g;)
(∵ Change the dummy index n→ l for the second and fourth terms)
= ( ∇2 H(0) ) i;g; . (35)

heading: C.4 JACOBIAN AND HESSIAN OPERATORS
text: Considering a similar discussion, we can show the following dependencies. For the Jacobian model, J [H(1)] ∈ R|V|×f×d×d,
( J [H(1)] ) i;g;kl =
( ∂ H(1)
∂xl ) i;g;k
(36)
= ∑ j Dij;;l(H (1) j;g;k − H (1) i;g;k) (37)
= (D̃ ⊗ H(1))i;g;lk. (38)
For the Hessian model, Hess[H(0)] ∈ R|V|×f×d×d,( Hess[H(0)] ) i;g;kl = ( ∂
∂xk
∂
∂xl H(0) ) i;g;
(39)
= ∑ j,m Dij;;k[Djm;;l(H(0)m;g; − H(0)l;g;)− Dim;;l(H(0)m;g; − H (0) i;g;)] (40)
= [ (D̃⊗ D̃) ∗ H(0) ] i;g;kl . (41)
D ISOGCN MODELING DETAILS

heading: To achieve isometric transformation invariance and equivariance, there are several rules to follow.
text: Here, we describe the desired focus when constructing an IsoGCN model. In this section, a rank-p tensor denotes a tensor the rank of which is p ≥ 1 and σ denotes a nonlinear activation function. W is a trainable weight matrix and b is a trainable bias.

heading: D.1 ACTIVATION AND BIAS

heading: As the nonlinear activation function is not isometric transformation equivariant, nonlinear activation
text: to rank-p tensors cannot be applied, while one can apply any activation to rank-0 tensors. In addition, adding bias is also not isometric transformation equivariant, one cannot add bias when performing an affine transformation to rank-p tensors. Again, one can add bias to rank-0 tensors.
Thus, for instance, if one converts from rank-0 tensors H(0) to rank-1 tensors using IsoAM G, G∗σ(H(0)W+b) and (G∗σ(H(0)))W are isometric equivariant functions, however (G∗H(0))W+b and σ ( (G ∗ σ(H(0)))W ) are not due to the bias and the nonlinear activation, respectively. Like-
wise, regarding a conversion from rank-1 tensors H(1) to rank-0 tensors, σ ( (G H(1))W + b ) and
σ ( G (H(1)W ) )
are isometric transformation invariant functions; however, G (H(1)W + b) and (G σ(H(1)))W + b are not. To convert rank-p tensors to rank-q tensors (q ≥ 1), one can apply neither bias nor nonlinear activation. To add nonlinearity to such a conversion, we can multiply the converted rank-0 tensors σ(( ⊗p G H(p))W + b) with the input tensors H(p) or the output tensors H(q).

heading: D.2 PREPROCESSING OF INPUT FEATURE
text: Similarly to the discussion regarding the biases, we have to take care of the preprocessing of rank-p tensors to retain isometric transformation invariance because adding a constant array and component-wise scaling could distort the tensors, resulting in broken isometric transformation equivariance.
For instance, H(p)/Stdall [ H(p) ] is a valid transformation to retain isometric transformation equiv-
ariance, assuming Stdall [ H(p) ] ∈ R is a standard deviation of all components of H(p). However,
conversions such as H(p)/Stdcomponent [ H(p) ] and H(p) − Mean [ H(p) ] are not isometric trans-
formation equivariant, assuming that Stdcomponent [ H(p) ] ∈ Rdp is a component-wise standard
deviation.

heading: D.3 SCALING
text: Because the concrete instance of IsoAM D̃ corresponds to the differential operator, the scale of the output after operations regarding D̃ can be huge. Thus, we rescale D̃ using the scaling factor
[ Meansample,i(D̃2ii;;1 + D̃ 2 ii;;2 + D̃ 2 ii;;3) ]1/2 , where Meansample,i denotes the mean over the samples
and vertices.
D.4 IMPLEMENTATION
Because an adjacency matrix A is usually a sparse matrix for a regular mesh, A(m) in equation 16 is also a sparse matrix for a sufficiently small m. Thus, we can leverage sparse matrix multiplication in the IsoGCN computation. This is one major reason why IsoGCNs can compute rapidly. If the multiplication (tensor product or contraction) of IsoAMs must be computed multiple times the associative property of the IsoAM can be utilized.

heading: For instance, it is apparent that
text: [⊗k G]∗H(0) = G⊗(G⊗. . . (G∗H(0))). Assuming that the number
of nonzero elements in A(m) equals n and H(0) ∈ R|V|×f , then the computational complexity of the right-hand side is O(n|V|fdk). This is an exponential order regarding d. However, d and k are usually small numbers (typically d = 3 and k ≤ 4). Therefore one can compute an IsoGCN layer with a realistic spatial dimension d and tensor rank k fast and memory efficiently. In our implementation, both a sparse matrix operation and associative property are utilized to realize fast computation.

heading: E.1 MODEL ARCHITECTURES

heading: E EXPERIMENT DETAILS: DIFFERENTIAL OPERATOR DATASET
text: Figure 4 represents the IsoGCN model used for the differential operator dataset. We used the tanh activation function as a nonlinear activation function because we expect the target temperature field to be smooth. Therefore, we avoid using non-differentiable activation functions such as the rectified linear unit (ReLU) (Nair & Hinton, 2010). For GCN and its variants, we simply replaced the IsoGCN layers with the corresponding ones. We stacked m (= 2, 5) layers for GCN, GIN, GCNII, and Cluster-GCN. We used an m hop adjacency matrix for SGCN.
For the TFN and SE(3)-Transformer, we set the hyperparameters to have almost the same number of parameters as in the IsoGCN model. The settings of the hyperparameters are shown in Table 5.

heading: E.2 RESULT DETAILS
text: Table 6 represents the detailed comparison of training results. The results show that an IsoGCN outperforms other GCN models for all settings. Compared to other equivariant models, IsoGCN has competitive performance compared to equivariant models with shorter inference time as shown in Table 7. Therefore, it can be found out the proposed model has a strong expressive power to express differential regarding space with less computation resources compared to the TFN and SE(3)Transformer.

heading: F EXPERIMENTS DETAILS: ANISOTROPIC NONLINEAR HEAT EQUATION DATASET

heading: F.1 DATASET
text: The purpose of the experiment was to solve the anisotropic nonlinear heat diffusion under an adiabatic boundary condition. The governing equation is defined as follows:
Ω ⊂ R3, (42) ∂T (x, t)
∂t = ∇ ·C(T (x, t))∇T (x, t), in Ω, (43)
T (x, t = 0) = T0.0(x), in Ω, (44) ∇T (x, t)|x=xb · n(xb) = 0, on ∂Ω, (45)
where T is the temperature field, T0.0 is the initial temperature field, C ∈ Rd×d is an anisotropic diffusion tensor and n(xb) is the normal vector at xb ∈ ∂Ω. Here, C depends on temperature thus the equation is nonlinear. We randomly generate C(T = −1) for it to be a positive semidefinite symmetric tensor with eigenvalues varying from 0.0 to 0.02. Then, we defined the linear temperature dependency the slope of which is −C(T = −1)/4. The function of the anisotropic diffusion tensor is uniform for each sample. The task is defined to predict the temperature field at t = 0.2, 0.4, 0.6, 1.0 (T0.2, T0.4, T0.6, T0.8, T1.0) from the given initial temperature field, material property, and mesh geometry. However, the performance is evaluated only with T1.0 to focus on the predictive performance. We inserted other output features to stabilize the trainings. Accordingly, the diffusion number of this problem is C∆t/(∆x)2 ' 10.04 assuming ∆x ' 10.0−3. Figure 5 represents the process of generating the dataset. We generated up to 9 FEA results for each CAD shape. To avoid data leakage in terms of the CAD shapes, we first split them into training, validation, and test datasets, and then applied the following process.
Using one CAD shape, we generated up to three meshes using clscale (a control parameter of the mesh characteristic lengths) = 0.20, 0.25, and 0.30. To facilitate the training process, we scaled the meshes to fit into a cube with an edge length equal to 1.
Using one mesh, we generated three initial conditions randomly using a Fourier series of the 2nd to 10th orders. We then applied an FEA to each initial condition and material property determined randomly as described above. We applied an implicit method to solve time evolutions and a direct method to solve the linear equations. The FEA time step ∆t was set to 0.01.
During this process, some of the meshes or FEA results may not have been available due to excessive computation time or non-convergence. Therefore, the size of the dataset was not exactly equal to the number multiplied by 9. Finally, we obtained 439 FEA results for the training dataset, 143 FEA results for the validation dataset, and 140 FEA results for the test dataset.
Published as a conference paper at ICLR 2021
<latexit sha1_base64="hOssCaZXMVf8qIaFIWJe17EJ8Cw=">AAAEnnicjVNdb9MwFPW2AKN8bINHXiyqSjxAaapJ4wGhwRjiBTEk2lU0UeU4TmvVdoLtMCov/Ape4X/xb7CTQNsUpFlKdH3vPedcH8tRxqjSvd6vre0d79r1G7s3W7du37m7t39wb6jSXGIywClL5ShCijAqyEBTzcgokwTxiJHzaH7i6udfiFQ0FR/1IiMhR1NBE4qRtqlPgSZftTl5+bqY7Ld73V654Gbg10Eb1OtscrDzLYhTnHMiNGZIqbHfy3RokNQUM1K0glyRDOE5mpKxDQXiRIWmHLmAHZuJYZJK+wkNy+wqwiCu1IJHtpMjPVPNmkv+sxbxhrJOnoWGiizXROBKOMkZ1Cl0fsCYSoI1W9gAYUnt7BDPkERYW9dagSAXOOUcidgEMU2SYtwPTZDYuolh2y/cv18UEHZSGVOBpCNKEiKtKxSxdYKswRBkzizESqLlxvKt46j4C3Si8CksVa0oFU/czcMriFsLlvJLYUu2qrzC+Sf7f8pcUF2MfcsHA3cTkptx2w9LO1wNXlA9g+pzjiSBkT3xnOiGp0hXAzGS6K4bJ5B0OtOXE7NpQ0yY9buRHFjnIm4GzeZhlR/afAeuVUhVIZsuZzJdTvPcTvPYWlJP9KLJj2R19Prgbl/iArM8RtDUOF3DnF4BgSJVYVzn5YpD7sX6zfe5GQz7Xf+we/ThsH38qn67u+ABeAgeAR8cgWPwFpyBAcBAgO/gB/jpQe+N9857X7Vub9WY+2BteaPf5fiNmQ==</latexit>
<latexit sha1_base64="yVdg73GTKvaP/vOenFWpBOAWlhk=">AAAEn3icjVNdb9MwFPW2AKN8bINHXgzVJB6gNNWk8YDQBJrgBVQk2g0lUeU4N61Vxwm2w6iy8C94hd/Fv8FOAm1TkGYp0fW995xzfSyHGWdK9/u/trZ3nGvXb+ze7Ny6fefu3v7BvbFKc0lhRFOeyvOQKOBMwEgzzeE8k0CSkMNZOH9t62dfQCqWio96kUGQkKlgMaNEm5Tna/iqi3egZuVkv9vv9auFNwO3CbqoWcPJwc43P0ppnoDQlBOlPLef6aAgUjPKoez4uYKM0DmZgmdCQRJQQVHNXOJDk4lwnErzCY2r7CqiIIlSiyQ0nQnRM9Wu2eQ/a2HSUtbx86BgIss1CFoLxznHOsXWEBwxCVTzhQkIlczMjumMSEK1sa3jC7igaZIQERV+xOK49AZB4cemXkS465b2PyhLjA9TGTFBpCWKY5DGFUb4OkHWYvAzaxbhFdFyY/jWcUz8BVpR/AxXqkaUiaf26vEVxI0FS/mlsCFbVV7h/JP9P2UumC491/Bh396ETAqv6waVHbaGL5ieYfU5JxJwaE48B93ylOh6IA6x7tlxfMmmM305KTZtiIAbv1vJkXEuTIpRu3lc58cmf4jXKlBXYNPlTKbLaV6YaZ4YS5qJXrb5iayP3hzc7iucXyyP4bc1Ttcwp1dAkFDVGNt5ueKQfbFu+31uBuNBzz3qHX846p68at7uLnqAHqHHyEXH6AS9RUM0QhSl6Dv6gX46D503zntnWLdubzWY+2htOZ9+A0xKjmg=</latexit>
<latexit sha1_base64="xM3z+uAOq8tPJNgckAHYO0pk7Qg=">AAAEqHicjVNdb9MwFPW2AKN8deORF4uqEkJQmmrSeEBogIp4LBJth5qqcpyb1qrjBNthq7LwU3iFv8S/wU4CbVOQZinR9b33nHN9EvsJZ0p3u7/29g+cGzdvHd5u3Ll77/6D5tHxSMWppDCkMY/luU8UcCZgqJnmcJ5IIJHPYewv39n6+CtIxWLxSa8SmEZkLljIKNEmNWseexoudfa+/wZLUCnXKp81W91Ot1h4N3CroIWqNZgdHXzzgpimEQhNOVFq4nYTPc2I1IxyyBteqiAhdEnmMDGhIBGoaVYMn+O2yQQ4jKV5hMZFdhORkUipVeSbzojoharXbPKfNT+qKevw5TRjIkk1CFoKhynHOsbWGRwwCVTzlQkIlczMjumCSEK18a/hCbigcRQREWRewMIwn/SmmReaehbglpvbdy/PMW7HMmCCSEsUhiCNK4zwbYKkxuAl1izCC6L1xvBt45j4C7Si+AUuVI0oE8/tP4CvIW4sWMuvhQ3ZpvIG55/s/ylTwXQ+cQ0f9uyXkFE2abnTwg5bwxdML7D6khIJ2DcnXoKueUp0ORCHUHfsOJ5k84W+mmW7NgTAjd+15NA450fZsN48KvMjk2/jrQqUFdh1OZHxeppXZppnxpJqotd1fiLLo1cHt/sC52XrY3h1jf4Wpn8NBPFVibGdVxsO2Rvr1u/nbjDqddyTzunHk9bZ2+ruHqJH6DF6glx0is7QBzRAQ0TRJfqOfqCfzlNn4Iydz2Xr/l6FeYi2luP/Bo0PkVA=</latexit>
<latexit sha1_base64="FIcp+4dBaDbsiDoquO/TI+LHhDY=">AAAEq3icjVNdb9MwFPW2AqN8dSCeeLGoJvHASlMNjQdAE2gSj0PQbqKJKse5aa06TrAdRuWF/8Ir/CL+DXYSaJuCNEuJru+955zrkzjMOFO63/+1tb3Tunb9xu7N9q3bd+7e6+zdH6k0lxSGNOWpPA+JAs4EDDXTHM4zCSQJOZyF87eufvYFpGKp+KgXGQQJmQoWM0q0TU06D30NX7WhXFHCocCvcL83eD7pdPu9frnwZuDVQRfV63Syt/PNj1KaJyA05USpsdfPdGCI1Ixa3rafK8gInZMpjG0oSAIqMOX8Bd63mQjHqbSP0LjMriIMSZRaJKHtTIieqWbNJf9ZC5OGso5fBIaJLNcgaCUc5xzrFDtzcMQkUM0XNiBUMjs7pjMiCdXWwrYv4IKmSUJEZPyIxXExHgTGj23dRLjrFe49KAqM91MZMUGkI4pjkNYVRvg6QdZg8DNnFuEl0XJj+dZxTPwFOlH8DJeqVpSJA/cb4CuIWwuW8kthS7aqvML5J/t/ylwwXYw9y4d99yVkYsZdLyjtcDV8wfQMq885kYBDe+I56IanRFcDcYh1z43jSzad6cuJ2bQhAm79biSH1rkwMcNm86jKj2x+H69VoKrApsuZTJfTvLTTPLWW1BO9bvITWR29PrjblzjfLI/hNzVO1jAnV0CQUFUY13m54lBhb6zXvJ+bwWjQ8w57R+8Pu8dv6ru7ix6hx+gJ8tAROkbv0CkaIooM+o5+oJ+tg9aH1qeWX7Vub9WYB2htteA3jrGRiQ==</latexit>
<latexit sha1_base64="y7fiJ4jHEByvjPZGoa23Gz1K2TY=">AAAEq3icjVNdb9MwFPW2AKN8rAPxxItFNYkHVpJq0ngANIEm8TgE7SaaqnKcm9aq4wTbYVRe+C+8wi/i32AnhbYpSLOU6Pree865PomjnDOlff/X1vaOd+Pmrd3brTt3793fa+8/GKiskBT6NOOZvIiIAs4E9DXTHC5yCSSNOJxHs7eufv4FpGKZ+KjnOYxSMhEsYZRomxq3H4UavmpDuaKEQ4lfYb/b88ftjt/1q4U3g2ARdNBinY33d76FcUaLFISmnCg1DPxcjwyRmlHL2woLBTmhMzKBoQ0FSUGNTDV/iQ9sJsZJJu0jNK6yqwhDUqXmaWQ7U6KnqllzyX/WorShrJMXI8NEXmgQtBZOCo51hp05OGYSqOZzGxAqmZ0d0ymRhGprYSsUcEmzNCUiNmHMkqQc9kYmTGzdxLgTlO7dK0uMDzIZM0GkI0oSkNYVRvg6Qd5gCHNnFuEV0XJj+dZxTPwFOlH8HFeqVpSJQ/cb4GuIWwuW8kthS7aqvML5J/t/ykIwXQ4Dy4dD9yVkaoadYFTZ4Wr4kukpVp8LIgFH9sQz0A1Pia4H4pDorhsnlGwy1Vdjs2lDDNz63Uj2rXNRavrN5kGdH9j8AV6rQF2BTZdzmS2neWmneWYtWUz0uslPZH30xcHdvsKFZnmMsKlxuoY5vQaCRKrGuM6rFYdKe2OD5v3cDAa9bnDUPX5/1Dl5s7i7u+gxeoKeogAdoxP0Dp2hPqLIoO/oB/rpHXofvE9eWLduby0wD9Ha8uA3ebyRhA==</latexit>
<latexit sha1_base64="4YDtS8ECjugI70StCFvN3VYgdPI=">AAAEq3icjVNdb9MwFPW2AKN8dSCeeLGoJvHASlImjQdAE2gSj0PQbqKpKse5aa06TrAdRuWF/8Ir/CL+DXZSaJuCNEuJru+955zrkzjKOVPa939tbe94167f2L3ZunX7zt177b37A5UVkkKfZjyT5xFRwJmAvmaaw3kugaQRh7No9tbVz76AVCwTH/U8h1FKJoIljBJtU+P2w1DDV20oV5RwKPEr7Hef++N2x+/61cKbQbAIOmixTsd7O9/COKNFCkJTTpQaBn6uR4ZIzajlbYWFgpzQGZnA0IaCpKBGppq/xPs2E+Mkk/YRGlfZVYQhqVLzNLKdKdFT1ay55D9rUdpQ1smLkWEiLzQIWgsnBcc6w84cHDMJVPO5DQiVzM6O6ZRIQrW1sBUKuKBZmhIRmzBmSVIOeyMTJrZuYtwJSvfulSXG+5mMmSDSESUJSOsKI3ydIG8whLkzi/CKaLmxfOs4Jv4CnSh+hitVK8rEgfsN8BXErQVL+aWwJVtVXuH8k/0/ZSGYLoeB5cOh+xIyNcNOMKrscDV8wfQUq88FkYAje+IZ6IanRNcDcUh0140TSjaZ6sux2bQhBm79biT71rkoNf1m86DOD2x+H69VoK7Apsu5zJbTvLTTPLWWLCZ63eQnsj764uBuX+FCszxG2NQ4WcOcXAFBIlVjXOflikOlvbFB835uBoNeNzjsHr0/7By/WdzdXfQIPUZPUICO0DF6h05RH1Fk0Hf0A/30DrwP3icvrFu3txaYB2htefAbfe6RhQ==</latexit>
<latexit sha1_base64="msSuJfcNH32OWDGTHyF2HxMHjQc=">AAAErnicjVNdb9MwFPW2AKN8rIMXJF4sqko8QEmqSeMBoQk0Cd6GRLtJTVQ5jtNatZ1gO4zKy34Nr/B/+DfYSaBtCtIsJbq+H+fce2zHOaNK+/6vnd0979btO/t3O/fuP3h40D18NFZZITEZ4Yxl8iJGijAqyEhTzchFLgniMSPn8eK9i59/JVLRTHzWy5xEHM0ETSlG2rqm3SehJt+0+SiopohBnImEukg57fb8gV8tuG0EjdEDzTqbHu5dh0mGC06ExgwpNQn8XEcGSU0xI2UnLBTJEV6gGZlYUyBOVGSqEUrYt54Eppm0n9Cw8q5XGMSVWvLYZnKk56odc85/xmLeYtbp68hQkReaCFwTpwWDOoNOH5hQSbBmS2sgLK0WGOI5kghrq2InFOQSZ5wjkZgwoWlaToaRCVMbNwnsBaX7D8sSwn4mEyqQdEBpSqRVxQq8CZC3EMLciWWPwQGtNhZvs46Kv4WOFL6CFaslpeKluwnwBuRWghX9itiCrTOvYf7x/h+ysLeonAQWD4buJCQ3k14QVXK4GLykeg7VlwJJAmM78YLolqZI1w0xkuqBayeUdDbXV1OzLUNCmNW75RxZ5WJuRu3kce0fW38fbkRIHSHbKucyW3XzxnbzwkrSdPS2jY9kPXozuNtXdaFZjRG2OU43ak5vUIFiVde4zKs1hdyLDdrvc9sYDwfB0eD401Hv5F3zdvfBU/AMPAcBOAYn4AM4AyOAwTX4Dn6An57vjb3Im9apuztNzWOwsbz5bwJKlD8=</latexit>
<latexit sha1_base64="lHk8XfLKuKImx0Dx7SJ2T8cRysY=">AAAE2nicjVPPb9MwFPa2AGP86uC4i0VViQOUppo0DghNoElckIZEu0lNVDnOy2rVcYLtMJUsB7ghrvwPu8Kfw3+DnWS0SUGapUQv3/P7vve+2EHKmdKDwe+NzS3nxs1b27d37ty9d/9BZ/fhWCWZpDCiCU/kaUAUcCZgpJnmcJpKIHHA4SSYv7H5k08gFUvEB71IwY/JmWARo0QbaNrZ82KiZzLO3xENkhHu4VQmKUi9KKad7qA/KBdeD9w66KJ6HU93ty69MKFZDEJTTpSauINU+zmRmlEOxY6XKUgJnZMzmJhQkBiUn5dTFLhnkBBHiTSP0LhEVytyEiu1iAOz0/as2jkL/jMXxC1lHb3wcybSTIOglXCUcawTbC3CIZNANV+YgFDJTO+Yzogk1DjUZNZs/rnBnVt1nSRcGVjAOU3imIgw90IWRcVk6OdeZIjyEHfdwr6HRYFxL5EhE0RaxSgCaewzf6JJkLYYvNS6SnhJtPwwfM06Jv4WWlH8HJeqRpSJZ/bU4GuIG6+W8kthQ7aqvMJ5hf6fMhNMFxPX8OGrIzjpun5ph83hc6ZnWH3MiAQcmInnoFWTguiqIQ6R7tt2PMnOZvpimq/bEAI3frfAkXEuiPNRe/O4wscG7+FGBqoMrLtsbs2ym5emm6fGkrqjV21+IqvR68Htd1nn5csxvLbGUaPm6BoVJFBVjd15seKQvdpu+yKvB+Nh393vH7zf7x6+ri/5NtpDj9ET5KIDdIjeomM0QhR9QZfoJ/rleM5X55vzvdq6uVHXPEKN5fz4A+IopyE=</latexit>
F.2 INPUT FEATURES
To express the geometry information, we extracted the effective volume of the ith vertex V effectivei and the mean volume of the ith vertex V meani , which are defined as follows:
V effectivei = ∑ e∈N ei 1 4 Ve, (46)
V meani =
∑ e∈N ei Ve
|N ei | , (47)
where N ei is the set of elements, including the ith vertex. For GCN or its variant models, we tested several combinations of input vertex features T0.0, C, V effective, V mean, and x (Table 9). For the IsoGCN model, inputs were T0.0, C, V effective, and V mean.

heading: F.3 MODEL ARCHITECTURES
text: Figure 6 represents the IsoGCN model used for the anisotropic nonlinear heat equation dataset. We used the tanh activation function as a nonlinear activation function because we expect the target temperature field to be smooth. Therefore, we avoid using non-differentiable activation functions such as the rectified linear unit (ReLU) (Nair & Hinton, 2010). Although the model looks complicated, one propagation block corresponds to the first-order Taylor expansion T (t+∆t) ' ∇C ∇T (t)+T (t) because the propagation block is expressed as D̃ C MLP(T )D̃ ∗ T + T , where T denotes the
rank-0 tensor input to the propagation block. By stacking this propagation block p times, we can approximate the pth order Taylor expansion of the anisotropic nonlinear heat equation.
For GCN and its variants, we simply replaced the IsoGCN layers with the corresponding ones. We stacked m (= 2, 5) layers for GCN, GIN, GCNII, and Cluster-GCN. We used an m hop adjacency matrix for SGCN.
For the TFN and SE(3)-Transformer, we set the hyperparameters to as many parameters as possible that would fit on the GPU because the TFN and SE(3)-Transformer with almost the same number of parameters as in IsoGCN did not fit on the GPU we used (NVIDIA Tesla V100 with 32 GiB memory). The settings of the hyperparameters are shown in Table 8.

heading: F.4 RESULT DETAILS
text: Table 9 shows a detailed comparison of the training results. The inclusion of x in the input features of the baseline models did not improve the performance. In addition, if x is included in the input features, a loss of the generalization capacity for larger shapes compared to the training dataset may result as it extrapolates. The proposed model achieved the best performance compared to the baseline models considered. Therefore, we concluded that the essential features regarding the mesh shapes are included in D̃. Besides, IsoGCN can scale up to meshes with 1M vertices as shown in Figure 7.

Review summaries:
The paper proposes a network that operates on features of graphs that are embedded in a d-dim Euclidean space. The paper considers equivariance to a group G that is the direct product of permutations of N points and Euclidean transformations. The features they consider are tensor products of the N-dimensional natural representations of permutations and the d-dimensional standard representation of O(d). From the coordinates, an “isometric adjacency matrix” is created, which is such a tensor. This matrix is combined in various G-equivariant ways with the features and then linearly combined with learnable weights to create new features. These operations are interleaved with non-linearities to form the network. The authors compare to several graph network methods and show competitive performance on several tasks.