Equivariant Networks for Crystal Structures
Authors: Sékou-Oumar Kaba
Siamak Ravanbakhsh
Venue: NeurIPS 2022 Accept (2022)
Keywords: materials
deep learning
symmetry
equivariance
crystals
graph neural networks
geometric deep learning

Abstract: Supervised learning with deep models has tremendous potential for applications in materials science. Recently, graph neural networks have been used in this context, drawing direct inspiration from models for molecules. However, materials are typically much more structured than molecules, which is a feature that these models do not leverage. In this work, we introduce a class of models that are equivariant with respect to crystalline symmetry groups. We do this by defining a generalization of the message passing operations that can be used with more general permutation groups, or that can alternatively be seen as defining an expressive convolution operation on the crystal graph. Empirically, these models achieve competitive results with state-of-the-art on the Materials Project dataset.

Equivariant Networks for Crystal Structures
Sékou-Qumar Kaba, Siamak Ravanbakhsh
School of Computer Science,
McGill University
Mila Quebec Artificial Intelligence Institute
(kabaseko@mila,coo quebec, siamakēcs megill. ca}
Abstract
Supervised learning with deep models has tremendous potential for applications in materials science. Recently, graph neural networks have been used this context, drawing direct inspiration from models for molecules. However, materials are typically much more structured than molecules, which is feature that these models do not leverage. In this work, we introduce class of models that equivariant with respect to crystalline symmetry groups. We do this by defining generalization of the message passing operations that can be used with more general permutation groups, or that can alternatively be seen as defining an expressive convolution operation the crystal graph. Empirically, these models achieve competitive results with state-of-the-art on property prediction tasks.
Introduction
Deep learning has seen remarkable applications in computational chemistry, both on the side of molecular property prediction and molecule generation. For small molecules, these methods are
close to a level of precision that would make them suitable for practical applications [46]. However, less attention has been put to the problem of designing neural network architectures for the larger assemblies of atoms that constitute materials. This is crucial problem, as deep learning could advantageously complement computationally demanding ab initio simulation methods like DFT 28
In this paper. we address the problem of designing equivariant layers and use them for supervised learning on materials. We focus on crystals, materials characterized by the ordered arrangement of their atoms in lattices. Crystalline materials are largely present around us and essential in technological applications, as they include large number of metals, ceramics, and salts, among others [3] They can be described the regular repetition of a set of atoms, called the unit cell, in all directions of space. These patterns are analogous to wallpaper patterns, for which the repeated structure is an image. Crystals are characterized by high degree of symmetry which is fundamental understanding their physical properties. Consequently, we suggest that equivariant deep learning provides an appropriate framework to design function approximators for crystals.
Graph Neural Networks (GNNs) have recently been proposed for supervised prediction tasks on crystals [54] 70 [2]. We hypothesize here that these models may fall short in exploiting crystal symmetry and that structural information is lost when mapping a crystal structure particular, GNNs are equivariant to the permutation of atoms given as input to the model. This condition is overly restrictive and amounts forgetting about the ordered nature of a crystal. We propose to use models equivariant to product of groups GA x Sc. where GA acts at the level of the Bravais lattice, the underlying periodic grid of the crystal, and Se on unit cells. We show that our proposed equivariant architecture is more expressive than GNNs on this data structure.
graph. In
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

By using the crystal structure, our approach amounts to defining a group- equivaria convolution kernel on the crystal in a way that is completely analogous to convolutional neural networks (CNNs). This convolution is defined on a graph associated with a crystal structure. Our contributions are the following: 1) We derive different equivariant models based on the group-theoretical properties of crystals. 2) We show some of the limitations of GNNs for crystal data and propose an alternative data structure to be used with our architecture. 3) We perform a rigorous analysis, cleaning, and processing of the Materials Project database 3d and share the resulting processed dataset to serve as a benchmark for materials applications. 4) We perform experimental tests of our models on the Materials Project database and report results comparable or better than baselines.
2 Related works
Equivariant neural networks is well known that neural networks have to incorporate inductive biases to be useful in practice [67] 7 Using models that are invariant or equivariant to the symmetry of the data has proven to be particularly important inductive bias to promote generalization [8]. The first notable application of this idea was the CNN, for which each convolution layer is equivariant to translation 422. An alternative parameter-sharing view also appears in early works [57]; while more
recent equivariant networks have used both convolutional 1K
19], and parameter-sharing view A few notable symmetries considered in recent years are rotation in image and volumetric 651 permutation symmetry in sets
200
[5] 24
data
77
50 and graphs [3 :e. as well as Euclidean
660 10 and rotational symmetry 14
55
2
In this work, we build on foundational work on
hierarchical symmetries
45
The increasing availability of large materials datasets from high- makes deep learning more and more relevant for materials science. Following the successes of GNNs on molecular data, similar models have been proposed for materials as alternatives to methods based on feature engineering. Note that in what follows, we refer to GNNs in a general sense that includes message passing neural networks (MPNNs) 26 Many variants of GNNs exist 35
Deep learning for materials
throughput calculations T8
3
66
77
!e! the underlying idea being for each node to aggregate features of neighboring nodes in a permutation invariant, or in the case of [39] in an equivariant way. The CGCNN 770 and MEGNet models [12 rely on mapping crystal structures to graphs and applying GNNs to obtain prediction. For the SchNet model, the correspondence with graphs is less explicit (54] but still present. Other approaches combine permutation equivariance
with -quiivaiiae to design models for molecules and materials 5 5
3 Background on crystal symmetry
We first start by introducing some principles of crystallography that will be used to derive our main results. A more comprehensive treatment also be found in the references 20 59 whereas basics of group theory are covered in (52) for example.
Lattices A crystal can be described as the periodic and infinite repetition of pattern in all directions of space. Crystals are conveniently described using lattices as their underlying structures.
An n-dimensional lattice A can be defined as the set of integral combinations of the linearly indepen- dent lattice basis vectors a, E R.:
A Σ m,a; m. E Z
(I)
The lattice is entirely specified by its basis vectors a,. lattice is associated with a group of translations TA for which the multiplication rule is addition. This captures the translational symmetry of crystal. lattice also defines subsets of IR" called unit cells. These subsets have the property of tilling the space when translated by lattice vectors. Of particular importance is the primitive cell U for the basis a;:
U =
Σ a,a;
0 x <
(2)

---

In material, the unit cell comprises a set of atomic positions $ = x,) X, E U), where the integer Z, is the atomic number and x; the position of the atom. .S can contain an arbitrary number of atoms and must not possess particular structure. Together with the lattice A, the atomic positions provides a complete description of the crystal structure.
It is often useful to define the concept of sublattice. A sublattice Ap of A is lattice with basis vectors b, such that Ap A. Correspondingly, the supercell Cp is the unit cell associated with the sublattice Ap, for which Cp C U. The full lattice is generated by translations of the sublattice by set of centring vectors {0.... v. More formally, the sublattice associated with normal subgroup Тar of lattice translations, which specifies coset decomposition of the original lattice TA (0 TA,) U... (v, + TAp). The centring vectors are coset representatives with respect to that decomposition. It is clear that the centering vectors are also the set of lattice points contained in supercell Cp.
Example 3.1 (Graphene) In graphene, carbon atoms are arranged in a two-dimensional crystal with honeycomb structure. The underlying lattice A has basis vectors aı 2 [3 v3 and
a2 [3 -v3]. where a is a lattice constant. The set of atoms in the unit cell is {66 2
e [1 0I) (6, 2[1 }.
B
82
Up
b
Figure Graphene crystal structure
In Fig I blue dots represent carbon atoms. Black and white dots identify points of lattice A with unit cell U. Black points identify sublattice Ap defined by basis vectors b and b and with supercell Up. The centring vectors {0, a1} correspond to lattice points contained in the supercell.
Space groups Symmetry plays major role in the description of crystals. This symmetry is often directly visible through facets naturally occurring crystals. Mathematically, it described by space group G, the set of isometries that maps crystal structure to itself. As isometries, space groups are subgroups of the Euclidean group (n). A space group element can be described as tuple (W, t), where W is the linear part of the transformation and t translation. An element maps vector x E Rn to Wx + t.. The multiplication of space group elements is therefore given by (W1, t1) (W2, 12) (W, W., W2t, + 12). The point group P of a space group is the group obtained from linear part operations in G, which will in general be rotations and reflections.
Considering only elements in G' for which the linear part is identity (I, t). obtain the translation subgroup of the space group T. It is a normal subgroup, which allows defining the factor group G/T isomorphic to the point group P.
The crystallographic restriction theorem guarantees that only certain finite groups are valid point groups of space groups [17]. In particular, in 2 and 3 dimensions, only n-fold rotations with [2, 3, 4, 6) are allowed. Two space groups G and G' are said to belong to be of same type they can be related by change of coordinate system. There are 17 space group types in 2 dimensions and
230 in 3 dimensions [59]. For a lattice A, we call the Bravais group of the lattice PA the set of linear isometries that map A to itself. Bravais groups provide way classify lattices according their

---

symmetry. We say that two lattices A and A' belong to the same Bravais rype if their Bravais groups are the same matrix groups when written for suitable basis vectors of each lattice. The 5 Bravais types in 2 dimensions and the 14 in 3 dimensions are enumerated in the Appendix. The full symmetry group GA of lattice A is given by the semidirect product of its translation group with its Bravais group: CA T A PA-
Consider the space group crystal structure with lattice A and unit cell U. From the definition of crystal structure, it is clear that the translation subgroup of G will be Tλ. However, the point group of G will, in general, not be PA. This is because the unit cell may have less symmetry than the underlying lattice, which will result in P S PA (see example 3.2). This is the reason why the number of space groups much larger than the number of Bravais lattices.
Example 3.2 (Wallpaper pattern) The wallpaper in Figure 2 is described by a square Bravais lattice A and unit cell U with 4-fold rotational symmetry. The square lattice has symmetry group P4m, with D₂ Bravais group. However, the unit cell reduces the symmetry of the overall pattern since does not have reflection axes. The symmetry group of the wallpaper is, therefore, P4, with point group C4.
U
a
Figure 2: Egyptian wallpaper
Limitations of GNNs
In this section, we examine some of the limitations of GNNs on crystalline data, motivating our model by addressing these limitations. Typical models for crystalline data 77 12 will build graph from atoms in the unit cell and assign feature vector h, E R" to each. Edge features encoding distance might also be used. For convenience, this graph can be represented
sparse tensor H E N xn. where is the number of atoms considered, and which node and edge features are encoded on diagonal and off-diagonal entries, respectively. GNN is a function approximator f T:RN" xn - that produce a prediction, by successive application of layers 6: R N" xne + R xnei1, where & is the layer index.
Expressivity
GNNs are usually stack of layers that are permutation equivariant
((P @ H) = (P(g) @ P(g)) (H), Vg E Sv N*x
(3)
where P(g) is the permutation matrix associated with the group element While P(g) permutes nodes, P(g) & (g)) permutes the vectorized adjacency matrix. Our use of Kronecker product is due to the equality vec(PAPT) (P P))ec(A) for A E RNN.
This captures the fact that these models are designed to be insensitive to node ordering in H. The material is, in sense, treated if it was molecule. Much of the crystal structure is forgotten because it cannot be captured by ordering the nodes a specific way and is only encoded in positions or distances. We argue that permutation equivariance is an overly strong requirement and was mainly used for practical reasons. When possible, it should be more beneficial to use a model equivariant to the actual symmetry of the data; in this case, the crystal space groups, which will, in general, be much smaller than the symmetric group. Being equivariant to a smaller group results in less restrictive parameter sharing and more expressive model.
Note that this requirement different from that of (33)--quivariance characteristic of some archi- tectures 60 5 5 In these models, the E(n) group only acts on the position x; R part of each

---

atom's feature vector. By contrast, the symmetry group of crystal structure maps the crystal to itself, and its action a permutation. It therefore offers an alternative to building more powerful architectures for these systems that does not suppose that coordinate information is available. In particular, this approach is more suitable for abstract condensed matter systems, like spin and free- fermion lattice models, in which coordinates are not relevant. Moreover, space groups are subgroups of the Euclidean group; equivariance to space groups is thus less restrictive. In this work, we choose to concentrate on space group equivariance, but we still provide an (())-qquivarian version of our architecture, which is a straightforward extension of [53] in Appendix A..5 Note that equivariance (and not only invariance) is important, expressive invariant functions can be built by composing equivariant layers with an output pooling layer. Equivariant functions can also be used to predict local properties like magnetization and charge distribution for example.
Invariance and symmetry breaking Even the arrangement atoms a crystal symmetric, this does not have to carry over all local properties of the material. Spontaneous symmetry breaking is common in materials and crucial to describing phenomena such as magnetism and superconductivity Building graph only at the unit cell level does not allow to capture local properties that differ across unit cells. Using a supercell of multiple unit cells does not suffice to solve this problem since permutation equivariant models have the property that equal input elements will be mapped to equal outputs elements
41 [.
558
Input representation
Supercell Following the arguments of Section 4 we consider the set of atoms in a supereell instead of only in the unit cell, and add explicity symmetry breaking increase the representational power of the model. Since this increases the computational complexity of the method, we choose to keep the supercells small and define them with sublattice vectors b, 2a;- The supercell is therefore 8 times larger than the unit cell, with centring vectors E m;a, m: E 1). }.
For each atom, we build feature vector with a one-hot encoding the atomic number and a one-hot encoding identification of the unit cell it belongs to using the corresponding centering vector. We keep track of the index of cach atom within the unit cell, and the index of the atoms' unit cell ((a,)) = [onehot (Z,). onehot (a)], where E {1,.... 8} and E C, where the number of atoms in the unit cell. The encoding of the unit cell allows to break the symmetry between atoms mapped into each other by lattice translations.
Graph We construct a graph from atoms in the supercell, encoding relative distances between atoms as edge features. Inspired by [54], an edge feature vector is built from the distance between
atoms d, = ||x - Xyyl || as Cij = exp (-> (d.y - 14)²) The vector of Gaussian centers and are hyperparameters. We use this approach to facilitate comparaison previous works 54 MCC, although Bessel encondings 3d could also be considered. It can be seen as "soft" binning of interatomic distances. Using this approach, we use only distance features in contrast to methods that use position vectors. This has the benefit of simplicity while still allowing a complete description of the input structure 66 [3
Sparsity has proven to be useful inductive bias in graph representation learning 2 and it is also beneficial in reducing computational complexity. However, atomic bonds not unambiguously defined in crystals [6 M. We choose to follow a similar approach to (2] an edge is drawn between atoms if they share Voronoi face and if the distance between atoms is smaller than the sum of atomic Cordero radii plus cutoff A = 0.5A. This approach has the advantage of being physically sound and producing graphs that are relatively sparse. We provide more details on the graph-building strategy and compute metrics the resulting graphs in Appendix A.I
To preserve translational invariance for atoms at the boundary of the supercell, edges are initially also drawn to atoms outside the supercell. Then, if an edge points outside the supercell, its head is mapped to the corresponding representative node inside the supercell. This analogous to circular padding in image processing.
5

---

(a) Identification of the supercell. The sublattice
(b) Voronoi tessellation.
vectors are b; 2a.
(d) Periodization of the graph. Edges that point to nodes outside the supercell are mapped back to the corresponding atom inside the supercell. Markings show identical atoms.
(c) Drawing edges between atoms that share Voronoi faces. The additional condition related the Cordero radius not relevant here.
Figure 3: Building graph for the graphene crystal structure from Example 3.1
6 Equivariant crystal networks
Product groups Consider a crystal structure with space group G and in which each atom has a feature vector h; Then the space group acts permutation of the input atoms
g h (o)) Vg G,
where Ty and my are the permutations associated with group element g on the supercells and within the unit cells respectively.
If a dataset contains only samples that share the same crystal structure, then a model equivariant with respect to can readily be used. However, the case of dataset with multiple different crystal structures that would be used in a typical supervised learning setting is more challenging for two reasons. First, samples may have different space groups, which would require different models, each be trained with fraction of the data. Second, the group action may be different even for the same space group. This is because the unit cells may have different structures and numbers of atoms.
A solution to address these issues is to consider equivariance to a direct product of groups GA x Sc, where the symmetry group of the lattice GA acts across unit cells and Sc, the symmetric group, acts within unit cells:
(g. h))( (( Vg E GA,hE E
In this way, differences unit cell structures dealt with by the symmetric group, where parameter- sharing can handle variable-sized inputs (72). We still have to accommodate 14 different group actions GA corresponding to the different Bravais lattices. To avoid using a different model for each lattice, we propose two groups to deal with all the lattices. The first option is to consider the least symmetric Bravais lattice of primitive triclinic type and use its symmetry group PI TA Oz. This group is a subgroup of all the other lattice symmetry groups. The second option we consider to simply use the symmetric group SA that is the symmetric group across unit cells, which is an overgroup of all the lattice symmetry groups. This leaves us with a hierarchy of groups, with Sv. the symmetric group over all atoms of supercell, being the largest:
Pi x Sc S GA x Sc C SA x Sc S SN.
(6)
Pl-model
SA model
In our experiments, use the two groups in this hierarchy for different levels of expressivity.

---

Equivariant message passing Having defined the group action, we can now build the Equivariant Crystal Network (ECN). We will seek to use the message passing framework, which has demonstrated good performance on molecular data, and generalize it to obtain equivariance to other groups than the symmetric group. The update equations for message passing framework are
mij = &e (bf. e.))
(7)
m; = Σ m:ji
JEN.
h [+1 on (h), m +
The idea is define parameter-sharing patterns for functions Oe and di, such that there can be multiple versions while still retaining equivariance. Following [5I], we first define the parameter- sharing pattern of the set of input nodes N, with respect to group G the colored bipartite graph B), with the edge-color function a N x N + [1....,,, C.} and node-color function : fI.....a, .,C,. We also consider the action of the group G on edges ft as g (i.j) = (Tg (1) Tg (i) E G. We define the orbit G' (i,j) of edge (i.j) as the set of edges in which it can be moved to by the group action G. (i.j) = {g (i,j) 9 E G}. A similar definition applies to the orbit of node, G.1 ( (i) I G}.
Ω = (N, a,
We then make the following claim:
Claim 6.1 The layer defined by the Ce functions &e (635)) and the functions ((1)
m (a(t.))
(h', e.)),
(8)
mij+
(9)
JEN,
h On 3(6) (hi m!)
(10)
G-equivariant if the parameter-sharing pattern Q respects the equivariance condition:
a (1.)) = (k: = 8 - ( = jeG.i.
(1.5),
(11)
(12)
The proof of this claim follows in Appendix A.2 In words, the group action the graph creates node and edge orbits, and we use a different copy of dhe and dv, for each edge and node orbit, respectively. The computational process for producing the pattern is to find the orbit of G-action the edges (nodes) [51], and the computational cost of this orbit-finding process grows linearly with the number of edges (nodes) 30
This layer generalizes both MPNNS and equivariant multilayer perceptrons, such as CNNs. The MPNN is recovered with G = Sn and standard CNN with circular convolution with G Tx. 9(() ((.5) h and @k (h m.) ReLU (m,+1 +b()))
de (tt.f)
h/,,
We now consider a product group G x H, acting according to Eq. ( From Claim 1 of 664 the equivariant linear map for this group is the Kronecker product of equivariant maps for individual groups; see also 45 The reformulation for parameter-sharing patterns is the following. If parameters- sharing patterns Q and fh satisfy the equivariance condition for G' and H respectively, then the purameter-sharigg pattern = (N x MI, a, 3) satisfies the equivariant condition
a N x M M + a (a, (a= 6). 02 (i.j)).
x N....,C..2h
and
8: N: MI + (1, Cl. x (1,...,, (a, i) (B, (a) B2 (i)).
(13)
(14)
This simply means that new color is defined in the product pattern for each possible combination of colors in the original patterns. Example 6. demonstrates this idea with simple example.
7

---

aa(i.) We use MLPs to build functions de and 9(1) The functions used in the experiments are detailed in Appendix A.4 addition. we add a weighting factor to the edge aggregation 9 as this as been
shown to be beneficial by 770 53
m, Σ
miy. where
eij Sa. (myy)
(15)
JEN,
and de is simply linear layer. This change does affect the equivariance of the model.
Example 6.1 In our running example, the parameter-shane -hariinngg pattern for the message passing produced by the Kronecker product of the pattern for the P2 group (see Appendix A.3 for similar example with the P6m group) and the pattern for the symmetric group S2 as shown in the figure below (Top left). However, we only need to keep the colors for which there is corresponding edge (Top right). Message passing is used on resulting edge/node colored graph where similar colored nodes and edges use the same functions in message passing.
Figure Equivariant message passing for graphene
7 Model implementation
Network architecture We aimed to keep the architecture our ECN model as simple as possible. The model receives crystal structure graphs input, with one-hot encoded feature vectors for each atom. We use 128-dimensional embeddings and keep the same dimension for hidden layers throughout the network. The ECN consists of 6 layers of the equivariant message passing operation 6.1
This is followed by mean saanlnoooing operation over node embeddings of each graph and two-layer MLP that outputs the final prediction. Mean pooling is preferred other options because we only predict intensive physical quantities. These properties are "per-atom" and do not depend on the choice of supercell. Therefore, selecting pooling operation that respects this invariance makes sense. If we were to predict extensive quantities, sum pooling would be the preferred option.
Input features Many choices encode atomic features are possible and have been suggested in the literature. We perform experiments over these variants, For each atom, we encode its type a feature vector. We consider two strategies, using only information available from the periodic table. The first one is to use only the atomic number. The second one is to use the one-hot encoded group and period numbers for each atom. The second strategy could benefit from promoting better generalization since embeddings of atoms belonging to the same group or period will show a certain similarity. Around 24% of the structures retained in the Materials Project dataset have been computed using the Hubbard-U extension of DFT. Since this information can significantly influence the resulting it is added the atomic feature vector binary feature.
properties 61
Other details We implemented our model using Pytorch 49 of the equivariant passsage-pasiingg based on the Pytorch Scatter package [2) We use the AdamW
We use a sparse implementation

---

optimizer 43
with weight decay regularization. The full hyperparameter setup is provided in
Appendix A.7
8
Experiments
This standard dataset materials informatics comprises more than 120K materials with a complete specification of their crystal structure and some important physical properties obtained with high-throughput DFT calculations.
Materials Project We perform experiments using the Materials Project dataset 33
The Materials Project dataset has not been initially built to serve as machine learning benchmark; we perform some preprocessing to make it suitable. First, since multiple DFT calculations were sometimes performed on close initial configurations, some samples only show marginal structure differences and resulting properties. This is exemplified by the compound Li,MnO which appears 322 times in the database. Such duplicates can result in training-tess leakage. To prevent this, we consider two structures redundant if they have the same unit cell chemical formula and the same space group. Amongst a set of duplicate structures, the one with the lowest formation energy is chosen. Second, we filter out one-dimensional and two-dimensional materials from the dataset to only keep taree-dimensional materials. Finally, remove materials for which the unit cell contains more than 50 atoms. These are often associated with molecular and inorganic crystals with very different properties than the other materials. Training, validation, and test splits are 80%, 10%, and 10% of the dataset. We provide statistics on the processed dataset in Appendix A.6
Following previous work, we predict few relevant energetic properties: the formation energy E, the Fermi energy Ep, and the band gap Eg for the subset of insulating materials. We also predict the binary insulator or conductor character material. Finally, we also predict the magnetic moment per atom M. This can be seen as a graph regression or classification task. For regression, training is performed using the mean-squared error (MSE) loss function, but we report the mean-absolute error (MAE). For classification, we use the cross-entropy loss function.
Table 1: Results on the Materials Project dataset.
Property
Method
E(eV/too))
Ep (eV)
M (pa/utoo)
Eg (eV)
Metal precision
Nonmetal precision
Ours Original
CGCNN
0.039
0.363
0.388
80%
95%
MEGNET
0.028
0.33
78.9%
90.6%
SCHNET
0.041
CGCNN
0.048 +: 0.0002
0.307 + 0.001
0.111
= 0.001
0.399 + 0.006
81.2% +: 3.0
86.3% + 3.0
MEGNET
0.056 0.0002
0.365
0.007
0.110 0.001
0.434
0.006
72.1%
3.0
81.6%
BCN-PI
0.052 + 0.001
0.303 + 0.004
0.108
0.002
0.44 + 0.02
80%
4.0
84%
BCN-Sx
0.046 = 0.002
0,281 = 0.007
0.106
0.002
0.390 ++0002
79.8% + 2.0
83.2% + 1.0
Note that because these papers used different training and test splits and preprocessing schemes (even between themselves), our results cannot be directly compared. To alleviate that, we trained our own versions of two of the baselines using our splits and a similar training procedure. We obtain slightly better or comparable results to the baselines on all targets when evaluated on the same splits. The SA version offers better performance than the PI model overall, showing that it more beneficial lean on the side of having slightly more symmetry than necessary at the cost of some expressivity. We provide additional results for the model variants in Appendix [. The benefits of the increased expressivity on this task is in the not crucial, which we think can be explained in part by the relatively small size of the Materials Project dataset. In a larger data regime, we expect that the benefit increased expressivity will outweight the cost in generalization capability.
We compare the results obtained by our models to baselines 54
70
12
Perov-5 Finally, we perform experiments using the Perov-5 dataset as provided by (69 In this dataset, all the materials share the same Perovskite crystal structure. The task considered is the regression of the heat of formation computed through DFT. Results are shown on Table 2 The improvement on this dataset is significantly more important for the proposed model compared to the
We use version 2021.05 of the dataset

---

baselines than on the Materials Project dataset. We hypothesize that the fact that all the structures are shared in this dataset allows the model specialize more efficiently leading better generalization.
Conclusion
We have shown how to leverage crystal symmetry to build more expressive and physically motivated neural networks for materials data. This allows us obtain close equivalent of group equivariant convolution on this data structure. These models show excellent accuracy in supervised property prediction, which supports the idea that symmetry is a useful inductive bias. Such models could be used for other tasks materials such dynamics prediction, if the dynamics approximately preserves the crystal structure. We also think that these models have significant potential more abstract condensed matter systems such as spin models and free-fermion models. We have also defined equivariant message passing. generalization of the MPNN framework that can potentially be used on any data structures for which a group can capture the symmetry in sparse interactions between the basic elements. One limitation of this approach is that is not clear how to handle structures with different groups without using larger group like SA. A potential solution is drawing inspiration from the Natural Graph Neural Networks framework introduced in [27]. Another area of future improvement is on the computational efficiency of the equivariant message passing, which does not benefit from optimized algorithms available for convolutions.
Property Heat All
Method
CGCNN MEGNET ECN-SA
0.047 + 0.000
0.059 0.006
0.038 + 0.004
Table 2: Perov-5 results
Acknowledgments and Disclosure of Funding
We thank Mehran Shakerinava, Christopher Morris, Joey Bose, Simon Verret and the anonymous reviewers for their valuable comments. This project is part supported by the CIFAR Al chairs program and NSERC Discovery. S.-O. K.'s research is also supported by IVADO and the DeepMind Scholarship. Computational resources were provided by Mila and Compute Canada.
References
[1] Alvarez Santiago. cartography of the van der Waals territories Dalton Trans. 2013. 42. 8617-8636.
[2] Anderson Brandon, Hy Truong Son, Kondor Risi Cormorant: Covariant molecular neural networks // Advances in neural information processing systems. 2019. 32.
[3] Ashcroft Neil W, Mermin N David, others Solid state physics. 1976.
[4] Bartók Albert P. Kondor Risi, Csányi Gábor. On representing chemical environments // Physical Review B. 2013. 87, 18. 184115.
[5] Batzner Simon, Musaelian Albert, Sun Lixin, Geiger Mario, Mailoa Jonathan P., Kornbluth Mordechai, Molinari Nicola, Smidi Tess E., Kozinsky Boris. E(3)-Equivariant Graph Neural Networks for Data-Efficient and Accurate Interatomic Potentials. 2021.
[6] Beekman Aron, Rademaker Louk, Wezel Jasper van. An introduction to spontaneous symmetry breaking // SciPost Physics Lecture Notes. 2019. 011.
[7] Bengio Y., Courville A., Vincent Representation Learning: A Review and New Perspectives // IEEE Transactions on Pattern Analysis and Machine Intelligence. Aug 2013. 35, 8. 1798-1828.
[8] Bronstein Michael M, Bruna Joan, Cohen Taco, Velicković Petar. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges /| arXiv preprint urXiv:2104.13477 2021.
[9] Castelli Ivano E, Landis David D, Thygesen Kristian S, Dahl Søren, Chorkendorff lb, Jaramillo Thomas F, Jucobsen Karsten W. New cubic perovskites for one-and two-photon water splitting using the computational materials repository // Energy & Environmental Science. 2012. 5, 10. 9034-9043.
10

---

[10] Cesa Gabriele. Lang Leon, Weiler Maurice. A Program to Build E (N)-Equivariant Steerable CNNs Interationall Conference on Learning Representations. 2021.
[11] C Lowik, Das Abhishek, Goyal* Siddharth, Lavril* Thibaut, Shuaibi Muhammed, Riviere Morgane, Trun Revin, Herus-Domingo Javier, Ho Caleb, Hu Weihua, Palizhati Aini, Sriram Anuroop. Wood Brandon, Yoon Junwoong, Purikh Devi, Zitnick C. Lawrence, Ulissi Zachary. Open Catalyst 2020 (OC20) Datasct and Community Challenges // ACS Catalysis. 2021.
[12] Chen Chi, Ye Weike, Zuo Yunxing, Zheng Chen, Ong Shyue Ping. Graph Networks as a Universal Machine Learning Framework for Molecules and Crystals // Chemistry of Materials. 2019. 31, 9. -5722
[13] Cohen Taco, Welling Max. Group equivariant convolutional networks // International conference on machine learning. 2016. 2990-2999.
[14] Cohen Taco S., Geiger Mario, Köhler Jonas, Welling Max. Spherical CNNs // International Conference on Learning Representations. 2018.
[15] Cohen Taco S, Geiger Mario, Weiler Maurice. A general theory of equivariant cnns on homogeneous spaces // Advances in neural information processing systems. 2019. 32.
[16] Cordero Beatriz, Gómez Verónica, Platero-Prats Ana Revés Marc, Echeverría Jorge, Cre- mades Eduard, Barragán Flavia, Alvarez Santiago. Covalent radii revisited // Dalton Trans. 2008. 2832-2838.
[17] Coxeter H. S. M. Introduction to geometry. New York: John Wiley & Sons Inc., 1961. xvii+443.
[18] Curtarolo Stefano, Setyawan Wahyu, Hart Gus L. W., Jahnatek Michal, Chepulskii Roman V., Taylor Richard H., Wang Shidong, Xue Junkai, Yang Kesong, Levy Ohad, Mehl Michael J., Stokes Harold T., Demchenko Denis O., Morgan Dane. AFLOW: An automatic framework for high-throughput materials discovery // Computational Materials Science. 2012. 58. 218 226.
[19] Dehmamy Nima, Walters Robin, Liu Yanchen, Wang Dashun, Yu Rose. Automatic Symme- try Discovery with Lie Algebra Convolutional Network // Advances in Neural Information Processing Systems. 2021. 34.
[20] Dresselhaus Mildred S. Dresselhaus Gene, Jorio Ado. Group theory: application to the physics of condensed matter. 2007.
[21] Esteves Carlos, Makadia Ameesh, Daniilidis Kostas. Spin-weighted spherical cnns // Advances in Neural Information Processing Systems. 2020. 33. 8614-8625.
[22] Fey Matthias. PyTorch Scatter. 2022.
[23] Finkelshtein Ben, Baskin Chaim, Maron Haggai, Dym Nadav. A simple and universal rotation equivariant point-cloud network // arXiv preprint arXiv:2203.01216. 2022.
[24] Finzi Marc, Welling Max, Wilson Andrew Gordon. A Practical Method for Constructing Equiv- ariant Multilayer Perceptrons for Arbitrary Matrix Groups // arXiv preprint arXiv:2104.09459 2021.
[25] Garg Vikas, Jegelka Stefanie, Jaakkola Tommi. Generalization and representational limits of graph neural networks // International Conference on Machine Learning. 2020. 3419-3430.
[26] Gilmer Justin, Schoenholz Samuel S., Riley Patrick F., Vinyals Oriol, Dahl George E. Neural Message Passing for Quantum Chemistry // Proceedings of the 34th International Conference on Machine Learning. 70. International Convention Centre, Sydney, Australia: PMLR, 06-11 Aug 2017. 1263-1272. (Proceedings of Machine Learning Research).
[27] Haan Pim de, Cohen Taco S, Welling Max. Natural Graph Networks Advances in Neural Information Processing Systems. 33. 2020. 3636-3646.

---

[28] Hasnip Philip J., Refson Keith. Probert Matt 1. J., Yates Jonathan R., Clark Stewart J., Pickard Chris J. Density functional theory the solid state // Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences. 2014, 372, 2011. 20130270.
[29] Hiß Gerhard, Holl Derek F, Newman Michael F. Computational Group Theory // Oberwolfach Reports. 2007. 3, 3. 1795-1878.
[30] Holt Derek F; Eick Bettina, O'Brien Eamonn A. Handbook of computational group theory. 2005.
[31] Horton Matthew Kristofer, Montoya Joseph Harold, Liu Miao, Persson Kristin Aslaug. High- throughput prediction of the ground-state collinear magnetic order of inorganic materials using Density Functional Theory // npj Computational Materials, jun 2019. 5, 1.
[32] Isayev Olexandr, Oses Corey, Toher Cormac, Gossett Eric, Curtarolo Stefano, Tropsha Alexan- der. Universal fragment descriptors for predicting properties of inorganic crystals // Nature Communications. Jun 2017. 1.
[33] Jain Anubhav, Ong Shyue Ping, Hautier Geoffroy, Chen Wei, Richards William Davidson, Dacek Stephen, Cholia Shreyas, Gunter Dan, Skinner David, Ceder Gerbrand, al. et. Commentary: The Materials Project: A materials genome approach to accelerating materials innovation // APL Materials. Jul 2013. 1, 1. 011002.
[34] Jones Owen, Bedford Francis, Waring J. B., Westwood J. O., Wyatt M. Digby. The Grammar of Ornament. London: Published by Day and Son, Lithographers to the Queen, Gate Street, Lincoln's 's Inn Fields, 1856.
[35] Kipf Thomas N., Welling Max. Semi- Superviid Classification with Graph Convolutional Networks // 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. 2017.
[36] Kirklin Scout Saal James E, Meredig Bryce, Thompson Alex, Doak Jeff W. Aykol Muratahan, Rühl Stephun, Wolverton Chris. The Open Quantum Materials Database (OQMD): assessing the accuracy of DFT formation energies // npj Computational Materials. 2015. 1. 15010.
[37] Klicpera Johannes, GroB Janek, Günneeaan Stephan. Directional Message Passing for Molec- ular Graphs // International Conference on Learning Representations, 2020.
[38] Kohn W., Sham L. J. Self-Consistent Equations Including Exchange and Correlation Effects // Phys. Rev. Nov 1965. 140. A1133-A1138
[39] Kondor Risi, Son Hy Truong, Pan Horace, Anderson Brandon, Trivedi Shubhendu. Covariant compositional networks for learning graphs // arXiv preprint arXiv: .02144. 2018.
[40] Kondor Risi, Trivedi Shubhendu. On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groups // Proceedings of the 35th International Conference on Machine Learning. 80. 10-15 Jul 2018. 2747-2755. (Proceedings of Machine Learning Research).
[41] Landau Lev Davidovich. On the theory of phase transitions. I. // Phys. Z. Sowjet. 1937. 11. 26.
[42] LeCun Yann, Bengio Yoshua, others Convolutional networks for images, speech, and time series The handbook of brain theory and neural networks. 1995. 3361, 10. 1995.
[43] Loshchilov Ilya, Hutter Frank. Decoupled Weight Decay Regularization International Conference on Learning Representations. 2019.
[44] Maron Haggai, Ben-Hamu Heli, Shamir Nadav, Lipman Yaron. Invariant and Equivariant Graph Networks // International Conference on Learning Representations. 2019.
[45] Maron Haggai, Litany Or Chechik Gal, Fetaya Ethan. On learning sets of symmetric elements // International Conference Machine Learning. 2020. 6734-6744.
12

---

[46] Noé Frank, Tkatchenko Alexandre, Müller Glaus-Robedt Clementi Cecilia. Machine Learning for Molecular Simulation // Annual Review of Physical Chemistry. 2020. 71, 361-390. PMID: 32092281.
[47] O' D'Mara Jordan, Meredig Bryce, Michel Kyle. Materials Data Infrastructure: A Case Study of the Citrination Platform to Examine Data Import, Storage, and Access // JOM. Jun 2016. 68, 8. 2031-20344
[48] Ong Shyue Ping, Richards William Davidson, Jain Anubhay, Hautier Geoffroy, Kocher Michael, Cholia Shreyas, Gunter Dan, Chevrier Vincent I., Persson Kristin A., Ceder Gerbrand. Python Materials Genomics (pymatgen); A robust, open-source python library for materials analysis I| Computational Materials Science. 2013. 68. 314 319.
[49] Paszke Adam, Gross Sam, Massa Francisco, Lerer Adam, Bradbury James, Chanan Gregory, Killeen Trevor, Lin Zeming, Gimelshein Natalia, Antiga Luca, others. Pytorch: An imperative style, nigh-performance deep learning library // Advances in neural information processing systems. 2019. 32. 8026-8037.
[50] Qi Charles R, Su Hao, Mo Kaichun, Guibas Leonidas J. Pointnet: Deep learning on point sets for 3d classification and segmentation Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.
[51] Ravanbakhsh Siamak, Schneider Jeff G., Póezos Barnabás. Equivariance Through Parameter- Sharing // ICML. 2017. 2892-2901.
[52] Rotman Joseph J. An introduction to the theory of groups. 148. 2012.
[53] Satorras Victor Garcia, Hoogeboom Emiel, Welling Max. E (n) equivariant graph neural networks // arXiv preprint arXiv:2102.09844, 2021.
[54] Schiitt Kristof, Kindermans Pieter-Jan, Felix Huziel Enoc Sauceda, Chmiela Stefan, Tkatchenko Alexandre, Müller Klaus-Robert, Schnet: continuous-file convolutional neural network for modeling quantum interactions // Advances in Neural Information Processing Systems. 2017. 991-1001.
[55] Shakerinava Mehran, Ravanbakhsh Siamak. Equivariant Networks for Pixelized Spheres // Proceedings of the 38th International Conference on Machine Learning. 139. 18-24 Jul 2021. 9477-9488. (Proceedings of Machine Learning Research).
[56] Shakerinava Mehran, Ravanbakhsh Siamak. Equivariant networks for pixelized spheres International Conference on Machine Learning. 2021. 9477-9488.
[57] Shawe- Taylor J. Building symmetries into feedforward networks // 1989 First IEE International Conference on Artificial Neural Networks, (Conf. Publ. No. 313). 1989. 158-162.
[58] Smidt Tess E., Geiger Mario, Miller Benjamin Kurt. Finding symmetry breaking order parame- ters with Euclidean neural networks Phys. Rev. Research. Jan 2021. L012002.
[59] Souvignier B. A general introduction to space groups // International Tables for Crystallography. 2016. 1.3, 22-41.
[60] Thomas Nathaniel, Smidt Tess, Kearnes Steven, Yang Lusann, Li Li, Kohlhoff Kai, Riley Patrick. Tensor field networks: Rotation-an and translation-equivert neural networks for 3d point clouds // arXiv preprint arXiv: 182.00890,, 2018
[61] Tolba Sarah A., Gameel Kareem M., Ali Basant A., Almossalami Hossam A., Allam Nageh The DFT+U: Approaches, Accuracy, and Applications // Density Functional Calculations. Rijeka: IntechOpen, 2018.
[62] Velicković Petar, Cucurull Guillem, Casanova Aranixa, Romero Adriana, Liô Pietro, Bengio Yoshua. Graph Attention Networks International Conference on Learning Representations. 2018.
13

---

[63] Villar Soledad. Hogg David W. Storey-Fisher Kate. Yao Weichi. Blum-Smith Ben. Scalars are universal: Equivariant machine learning, structured like classical physics Advances Neural Information Processing Systems. 2021.
[64] Wang Renhao, Albooyeh Marjan, Ravanbakhsh Siamak. Equivariant Maps for Hierarchical Structures. 2020.
[65] Weiler Maurice, Geiger Mario, Welling Max, Boomsma Wouter, Cohen Taco. 3d steerable cnns: Learning rotationally equivariant features in volumetric data // Advances in Neural Information Processing Systems. 2018. 10381-10392.
[66] Weyl Hermann. The Classical Groups. 1939.
[67] Wolpert D.H., Macready W.G. No free lunch theorems for optimization // IEEE Transactions on Evolutionary Computation. 1997. 1, 1. 67-82.
[68] Worrall Daniel E, Garbin Stephan J, Turnukhumbetov Daniyar, Brostow Gabriel J. Harmonic networks: Deep translation and rotation equivariance // Proceedings of the IEEE Conference Computer Vision and Pattern Recognition. 2017. 5028-5037.
[69] Xie Tian, Fu Xiang, Ganea Octavian-Eugen Barzilay Regina, Jaakkola Tommi. Crys- tal Diffusion Variational Autoencoder for Periodic Material Generation // arXiv preprint arXiv:21 10.06197. 2021.
[70] Xie Tian, Grossman Jeffrey C. Crystal Graph Convolutional Neural Networks for an Accurate and Interpretable Prediction of Material Properties Physical Review Letters. Apr 2018. 120, 14.
[71] Xu Keyulu, Hu Weihua, Leskovec Jure, Jegelka Stefanie. How Powerful are Graph Neural Networks? International Conference on Learning Representations. 2019.
[72] Zaheer Manzil, Kottrr Satwik, Ravanbakhsh Siamak, Poczos Barnabas, Salakhutdinov Russ R. Smola Alexander J. Deep Sets // Advances in Neural Information Processing Systems 30. 2017. 3391-3401.
[73] Zhang Yan, Zhang David W, Lacoste-Julien Simon, Burghouts Gertjan J., Snoek Cees G. M. Multiset- Equivarann Set Prediction with Approximate Implicit Differentiation // International Conference on Learning Representations. 2022.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] On both the theoretical and experimental side, the aligned with the abstract and introduction.
paper's contributions
(b) Did you describe the limitations of your work? [Yes] See Conclusion
(c) Did you discuss any potential negative societal impacts of your work? [N/A] We do not think that this is relevant to this work
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Theorem 6.1 and its proof in the Appendix A.2
(b) Did you include complete proofs of all theoretical results? [Yes] The only proof is included Appendix A.2
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)? [Yes] The details needed to reproduce the experiments included in the supplementary material Ap- pendix A.7 link to the relevant code will also be accessible upon publication of the paper.
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] The details needed to reproduce the experiments are included in the supplementary material Appendix AA77
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] This is provided in the supplementary material Appendix A.7
4. If you using existing assets (e.g., code, data, models) curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Introduction and Model implementation sections
(b) Did you mention the license of the assets? [Yes] This mentioned the supplementary material Appendix A.6
(c) Did you include any new assets either in the supplemental material or as URL? [No] The used dataset will be provided upon publication of the paper.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] This not relevant according to the license of the used assets.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] This relevant for this paper.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "YQr3kj8J_or", "reviewer": "Reviewer_os5D", "summary": "This paper studies bespoke equivariant neural architectures for processing crystal structures. Firstly, it is observed that crystals are discrete structures that exhibit substantially richer symmetries than the permutation group (which would normally used for abstract graph representations), which might allow for relaxed equivariance conditions, and hence, stronger architectures. Additionally, the authors show how to combine symmetry groups coming from various crystal structures within the same dataset, through the use of carefully constructed products of groups. The resulting equivariant architectures show promising results on a standard material science benchmark.", "questions": "Q1. Could you comment on whether there exist any other suitable baselines and/or benchmark datasets you could have used, provided you had more computational power at your disposal? \n\nQ2. Would you give any \"items of future work\" for interested authors who do have access to more computation?\n\nQ3. Could you provide at least 1-2 additional seeds for some of the models you evaluated, to get a feel for the variance?\n\nQ4. Does this paper, to the best of your knowledge, present the first rigorous exploration of geometric DL over crystal inputs, or are there any pieces of related work you would highlight? Some related works are already listed in the paper, but it is unclear if any of them are mindful of the symmetry structures you point out.", "limitations": "No concerns.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Originality**:\nFrom the methodological point of view, the proposed architectures and theoretical treatment appear like a direct application of the principles of geometric deep learning to the specific case of crystal symmetries. The idea of using the direct product of symmetry group is also in line with the GDL principles; see, for example, \"An equivariant Bayesian convolutional network predicts recombination hotspots and accurately resolves binding motifs\" from Brown & Lunter, which uses similar ideas to construct reverse-complement-symmetric architectures (for the processing of DNA).\n\nI do not have significant background in material science applications, therefore I cannot say for sure to what extent such ideas have already been explored in the context of crystals. However, the authors' aim to bring such symmetries into the spotlight, especially in light of naïve applications of graph neural networks to such problems, are certainly remarkable and deserve credit in either case.\n\n**Quality**:\nI find the proposed model to be solidly theoretically grounded, and the results to be reasonable and showing benefits against relevant GNN baselines. Of course, as before, I may not have sufficient context on what are the most meaningful architectures in this domain to use.\n\n**Clarity**:\nThis is one of the paper's absolute strongpoints. The use of guided examples (particularly in the case of graphene) are remarkably well-made and illustrate the points the paper makes perfectly. The paper was a delight to read as a result.\n\n**Significance**:\nThe main concern for me in the authors' evaluation is the lack of error bars, which the authors stated they are unable to provide due to computational resources. Would it be possible at least to get 1-2 additional seeds for the authors' proposal, in order to get some sense of the variance of the results? Otherwise, the results appear significant, but only one benchmark dataset is used (albeit, one that seems to be a standard choice for materials science).\n\n**Overall**:\nWhen weighing in on the different aspects of this paper's presentation, I tend to be in favour of accepting it. It presents a great application area for geometric deep learning, a principled investigation of how GDL can be applied in this space, and all of this is presented in a way that is very likely to reduce the barrier of entry into the field. I look forward to the release of the dataset and other artefacts.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "CxFrCgD1Pdk", "reviewer": "Reviewer_PzGa", "summary": "This paper generalizes the message passing operation which preserve $S_n$ equivariant to more general permutation groups, specifically crystalline symmetry groups, by parameter sharing pattern respecting equivariance condition. Models with the proposed message passing operation achieve comparable results with SOTA on the Materials Project dataset.\n\nThe idea is to improve the expressivity of GNNs by building an 8-time larger supercell and imposing parameter sharing using space group symmetry operations.\n\nIt takes me quite long time to review this paper, partly because I have less background of Crystal. I think this manuscript would be suitable for publication on journals like JCTC, rather than a machine learning conference.", "questions": "+ I am still a little confused what the authors mean by P1 and Sp symmetry group exactly. This is the symmetry of the P1 group: http://img.chem.ucl.ac.uk/sgp/large/002az1.htm. I don’t know what the authors mean by Sp group, which is not in the space group table. Example 6.1 is for a P2 group, which is different from P1 and Sp group.\n- For the experiments, the authors created a new dataset with 33971 crystals. They assume two structures are redundant if they have the same space group and chemical composition. This is problematic because different crystal structures can have both the same group and chemical composition.\n- For data preprocessing in Appendix A4, motivation of the last step is not clear. Why do they only include insulators?\n- The improvements with respect to SchNet and CGCNN are relatively small. The author didn’t consider latest methods like ALIGNN. Matbench provides a comparison between different models which could be used as a benchmark dataset (https://matbench.materialsproject.org/)", "limitations": "See above.", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n+ The motivation that full permutation equivariance might be overly restrictive is insightful and interesting.\n+ The empirical result is interesting.\n\nWeakness:\n+ The presentation is unclear. The paper could be more self-contained if “Voronoi face” and “Cordero radii” are explained in the main text or appendix. In line 199-209, there should be definition of $S_N$, $S_P$, $S_N$ and $S_{P×N}$. It seems that $S_N$ means permutations within unit cell, $S_P$ means permutations across unit cells (in supercell) derived from point group $P$, and $S_{P×N}$ means all permutations?\n+ Experiments are only conducted on one dataset, using different training and test splits and preprocessing scheme from previous works. The improvement is marginal according to line 301-302: “The $S_P$ version offers **slightly** better performance than the $P\\bar{1}$ model overall” – the differences comparing to previous work are similar to the different between two proposed variants in Table 1.\n+ As line 302-303 suggest, having more symmetry than necessary might be beneficial. The absence of ECN-“full permutation equivariance” in experiment drastically weaken the empirical result, i.e. the result cannot support the core proposition of this paper, that generalized permutation equivariance is better than full permutation equivariance (both input supercell) for crystal data.\n+ Novelty of the proposed method is limited. The proposed method could be viewed as an application of [49] Equivariance Through Parameter-Sharing for crystal symmetry group.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "2MLlNnD4tV", "reviewer": "Reviewer_MuHL", "summary": "The work introduces a method to design neural networks that are equivariant w.r.t. crystal symmetries, which arise in the description of materials structures. Given the success of symmetry as an inductive bias in molecular and materials science, based so far mostly on the groups SO(3), E(3), and the permutation group, the authors hypothesise that explicitly \"baking in\" equivariance w.r.t. crystal symmetries may provide a useful inductive bias for machine learning on crystals. First an introduction is given to a few basic concepts of solid-state physics such as lattices, point groups, and space groups based on the graphene example. Then the equivariant network architecture is described as a form of equivariant message passing. Equivariance is achieved by defining a weight-tying pattern that respects the symmetry of the crystal structure. Finally, the method is demonstrated on a series of targets available from Materials Project data. The method performs favorably. ", "questions": "1. \"In this paper, we address the problem of designing equivariant layers for supervised learning on materials.\" - why is this a problem? It is not clear a prior why equivariance (as opposed to invariance) is a desirable property as most target properties are invariant under the prescribed symmetries.\n\n2. The introduction could strongly benefit from some description of when crystal symmetries are important. E.g. there are many applications in computational materials science where crystal symmetry is not helpful (e.g. molecular dynamics, relaxations, ... ), but only the symmetries of E(3) as well as the symmetric group are still valid. On the other hand, current methods can *only* incorporate these and would not properly exploit additional symmetry information as proposed here. This should be laid out more clearly.\n\n3. Typo: \"tilling the space\" should likely be \"filling the space\"?\n\n4. I wonder if it might be beneficial to use a Bessel embedding instead of a Gaussian one? A number of works in the ML potential field have adopted this and it's been found to give improved performance, see Klicpera, J., Groß, J., & Günnemann, S. (2020). Directional message passing for molecular graphs. arXiv preprint arXiv:2003.03123. for example. \n\n5. \"This has the benefit of simplicity while still allowing a complete description of the input structure\" --> it's not clear why a Gaussian encoding vs just the interatomic distance is simpler, this sentence should be clarified. Distance is equally complete + simple, if those were the only 2 criterions (I realize it performs better in practice, but these are not the reasons why).\n\n6. Section 6.2 would benefit from some more detail. A lot of time is (rightfully) spent on group theory of crystals but then the actual core idea is introduced only briefly. This could be explained in more detail and ideally also giving some theoretical background. In particular, it is not clear how exactly the parameter-sharing is implemented: there is this sentence \"For our model, we choose functions \\phi_e^{\\alpha(i, j)} and \\phi_h^{\\beta(i, j)} to be MLPs with one hidden layer\", but it is not clear whether there is one separate MLP for each possible pair (i, j)? The appendix further confuses this by stating that \"the weights for the hidden layer and output layers are shared across all \\alpha(i, j)\" --> this presentation would *strongly* benefit from more detail, in particular how the parameter-sharing is implemented.\n\n7. In the comparison to other methods on MP: it is great that the authors do a proper filtering of the MP structures, this is solid data cleaning. However, when re-training another group's model there is often a chance that one underreports the performance of that model b/c the authors will have a better understanding of their own model's hyperparameters and a stronger incentive to optimize it. While this is in some sense difficult to avoid it would be great if the authors could at least clarify how the hyperparameters for the competing models were defined and if any hyperparameter optimization was performed and if yes, how that compared to the hyperparameter optimization done for ECN. \n\n8. The empirical results show only a small improvement. Given that no estimate of the statistical variance of the MAEs is given, it's difficult to say if these improvements are even statistically significant. If computationally feasible, this would *strongly* benefit from that. In addition, it would be great if the authors could add a (albeit speculative) sentence on why a model that explicitly accounts for crystal symmetry only marginally outperforms models that don't. \n\n9. In the conclusion the authors say \"Such models could be used for other tasks on materials such as dynamics prediction [...]\" --> this is difficult to imagine since any crystal symmetry would immediately be gone under molecular dynamics? Or was something different meant here? If yes, please specify more clearly.\n\n10. The authors at some point outline 2 strategies for encoding atoms, one based simply on a one-hot of the atomic number, one based on row+column of the PT and on a binary +U feature -- I could not find these experiments anywhere? Did I miss them?\n", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The paper provides an elegant solution to a non-trivial problem with potential applications in materials science. The solution to this problem is non-trivial and clean. The presentation of the mathematics as well as the required background in solid-state physics / crystal symmetry is well-developed using the classic graphene text book example and understandable even for an audience without a physics/materials science background. The experiment is being done with great care w.r.t. data set selection. \n\nWeaknesses: The paper only shows a single, simple experiment reporting average metrics on a common benchmark data set. While it shows marginal improvements over existing methods, the method provides no additional insights into specific materials or an attempt to discover new ones. It is not tested *in the wild* on a real materials problem. There is also no effect undertaken into understanding *why* the method works. Design choices are not tested or compared to others. The work would also strongly benefit from analyzing the inference time required on these networks. In particular the choice to construct a 2x2x2 supercell seems like it could rapidly lead to high inference cost. This leaves open many questions about whether such a system has potential for real-world impact. Finally, the presentation of the actual model leaves open many question. It is not immediately apparent *how* exactly the parameter-sharing MLPs are implemented. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept