Complete and Efficient Graph Transformers for Crystal Material Property Prediction
Authors: Keqiang Yan
Cong Fu
Xiaofeng Qian
Xiaoning Qian
Shuiwang Ji
Venue: ICLR 2024 poster (2024)
Keywords: crystal material property prediction
geometric graph representation learning
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)

Abstract: Crystal structures are characterized by atomic bases within a primitive unit cell that repeats along a regular lattice throughout 3D space. The periodic and infinite nature of crystals poses unique challenges for geometric graph representation learning. Specifically, constructing graphs that effectively capture  the complete  geometric information of crystals and handle chiral crystals remains an unsolved and challenging problem. In this paper, we introduce a novel approach that utilizes the periodic patterns of unit cells to establish the lattice-based representation for each atom, enabling efficient and expressive graph representations of crystals. Furthermore, we propose ComFormer, a SE(3) transformer designed specifically for crystalline materials. ComFormer includes two variants; namely, iComFormer that employs invariant geometric descriptors of Euclidean distances and angles, and eComFormer that utilizes equivariant vector representations.  Experimental results demonstrate the state-of-the-art predictive accuracy of ComFormer variants on various tasks across three widely-used crystal benchmarks. Our code is publicly available as part of the AIRS library (https://github.com/divelab/AIRS).

Published as a conference paper at ICLR 2024
COMPLETE AND EFFICIENT GRAPH TRANSFORMERS FOR CRYSTAL MATERIAL PROPERTY PREDICTION
Keqiang Yan, Cong Fu, Xiaofeng Qian', Xiaoning Qian', Shuiwang Ji
Texas A&M University College Station, TX 77843, USA
aegganngan, cong fu, xqian, feng, s ji .atamueedu
ABSTRACT
Crystal structures are characterized by atomic bases within primitive unit cell that repeats along a regular lattice throughout 3D space. The periodic and infinite nature of crystals poses unique challenges for geometric graph representation learning. Specifically, constructing graphs that effectively capture the complete geometric information of crystals and handle chiral crystals remains an unsolved and challeng- ing problem. In this paper, introduce novel approach that utilizes the periodic patterns of unit cells to establish the lattice-based representation for each atom, enabling efficient and expressive graph representations of crystals. Furthermore, we propose ComFormer, SE(3) transformer designed specifically for crystalline materials. ComFormer includes two variants; namely, iComFormer that employs in- variant geometric descriptors of Euclidean distances and angles, and eComFormer that utilizes equivariant vector representations. Experimental results demonstrate the state-of-ther of-the--rt--tt predictive accuracy of ComFormer variants on various tasks across three widely-used crystal benchmarks. Our code publicly available as part of the AIRS library (https: /ttp::////thhb///////g/ com/ com//ivveabbAARR.
INTRODUCTION
Accelerating the discovery of novel materials with desirable properties has values in many science. engineering, and biomedical applications (Ramprasad et al. [2017] Meredig et al. 2014 Olynnkk let al. 2016 Raccuglia et al. 2016; Ward et al. 2016 Liu et al. 2021b I Luo et al. 2021 Xie et al. current reliance on traditional, costly, and time-consuming
2022 Fu et 2023) Wang al. Chanussot al., [2021] Zhang et al. (023). However, the experimental methods poses practical challenges. In this regard, computational approaches based on quantum mechanies, such as density functional theory (DFT), have made significant contributions for predicting the physical and chemical properties of materials to guide materials discovery experiments. Nevertheless, the high computational cost associated with these methods (ranging from n to n², where represents the number of atoms) still poses hurdle for novel materials discovery the vast materials space.
To reduce computational cost, recent studies employ machine learning (MI.) models Yu et al. 2022 Ying et | 2021 Gao
2019 Liu et al. 2021a Fu et al. [022 Passaro & Zitnick [023] Gasteiger
al. 2020
Liao & Smidt 2022 Batzner et al. 2022 Satorras et al. 2021 Qiao al. 2022] Watson
ct al. 2022
Ahmad et al. 2018). such as crystal graph representations (Xie & Grossman 2018
Ling et al. 2023) with deep neural networks. However, these crystal graph representations have limitations in distinguishing different crystalline materials. In other words, they cannot guarantee capture the complete geometric information of input crystal structures, and may map different crystal structures with different properties to the same graph representation and produce the identical property predictions. Illustrative examples and detailed discussions can be found in Appendix A.
While graph representations that can capture any structural differences in small molecules have been investigated in previous works (Wang et al. 2022 Klicpera al. [202I) these methods fail short to capture periodic patterns of crystals and cannot maintain geometric completeness for crystals. Thus, how to construct graph representations for crystalline materials that can distinguish any different crystals and remain the same under crystal passive symmetries detailed in Sec. remains unsolved.
Equual senior authorship

---

Published as a conference paper at ICLR 2024
Establishing invariant (Anosova & Kurlin 2022 Edelsbrunner et al. 2021 and complete repre- sentations have also been investigated for other applications, especially representations of matrix forms consisting of aligned coordinates that can be used to measure the structural differences between crystals. Among them, AMD (Widdowson et al. 2022) and PDD (Widdowson & Kurlin 2022) are two recent works that satisfy continuity under small perturbations of atom positions by using Euclidean distances. However, AMD and PDD cannot distinguish chiral crystals and cannot be directly used to predict crystal properties without breaking the completeness.
In this work, we focus on crystal property prediction and propose SE(3) invariant and SO(3) equivari- ant crystal graph representations strive for geometric completeness for crystalline materials. Based on these expressive crystal graphs, we develop ComFormer with two variants; namely iComFormer that employs SE(3) invariant crystal graphs, and eComFormer that uses SO(3) equivariant crystal graphs. Both variants can scale to large-scale crystal datasets with complexity ((nk) where denotes the number of atoms in the crystal unit cell and denotes the average number of neighbors. We evaluate the proposed ComFormer variants on three widely-used crystalline material benchmarks. The state-of-the-art performances of ComFormer variants on various tasks across different scales demonstrate the effectiveness of the proposed crystal graphs and ComFormer variants.
2
BACKGROUND AND RELATED WORK
2.1 CRYSTAL STRUCTURES AND CRYSTAL PROPERTY PREDICTION
Crystal structures consist of a unit cell and a of atom bases associated with the unit cell. The unit cell is defined by a 3 x 3 lattice matrix indicating how the unit cell repeats itself in 3D space. Following the notations in Yan et al. (022)) crystal structures can be represented by M (A, P. L). where A- = a1. a2
an E R. X contains the da fd-dimensional feature vectors for n atoms in the unit cell, P P1. P2. Pr. E R contains the 3D Euclidean positions of n atoms in the unit cell, and the lattice matrix I. [e,, l2, Lgl E <R3x3 describes the repeating patterns of the unit cell structure in 3D space. Different from molecules and proteins, crystal structures naturally repeat infinitely in space. Formally, the infinite crystal structure can be described as
P (p.pp p. + k1 + k2l2 + kzLs, k2, ka E Z, Iilsisnn
(1)
À (d a,, 22isism m).
where the set P represents all 3D positions for every atom i in the repeated unit cell structure, and the set A represents the corresponding atom feature vector for every atom. In this paper, we aim to predict the target property value y. where y E R for regression tasks or y E {1,2, C} for classification tasks with C' classes by using crystal structure (A, P, L) as input.
2.2 GEOMETRIC COMPLETENESS FOR CRYSTAL GRAPHS AND CHALLENGES
Definition 1 (Geometrically Complete Crystal Graph). Following Widdowson & Kurlin (002)) ) crystal graph G is geometrically complete if G = G2 + M1 z M2, where S denotes that two crystals are isometric as defined Appendix AA.3 It means if two crystal graphs 91 and G2 are the same, the infinite crystal structures represented by Gi and G2 are identical.
A crystal graph G is considered geometrically complete if it can distinguish any minor structural differences between different crystalline materials.
Challenges. Achieving geometric completeness for crystals more challenging than small molecules. Different from molecules, crystals consist of unit cell structures and corresponding periodic pat- terns, with infinite number of atom positions described by P = {p. lp. = p: + k1l + kzl2 - kzL3, k1 l2 his E Z, E Z, S n). Previous methods for molecular representation learn- ing (Liu et al. 2022 Klicpera et al. 2021) only consider atom interactions within a radius, and cannot capture the periodic patterns of crystals. As a result, these methods fall short to maintain geometric completeness when extended to crystals
shown in Appendix A. Furthermore, different from rotation and translation invariances for molecular representation learning, unique crystal passive sym- metries arisen from enormous number of choices of unit cell structures and corresponding periodic patterns to describe a crystal structure need to be considered, including unit cell SE(3) invariance, unit cell SO(3) equivariance, and periodic invariance detailed in Sec. 3.1
2

---

Published as a conference paper at ICLR 2024
2.3
RELATED WORKS
Crystal graph neural networks. To capture atom interactions across cell boundaries, Xie & Grossman (0118) proposed CGCNN with multi-edge crystal graphs which use Euclidean distances as edge features. Based on the multi-edge crystal graph, Chen al. (2019) and Louis et al. (2020) proposed different model architectures to enhance property prediction accuracy. In addition to Euclidean distances, Choudhary
DeCost (202T) and Chen Ong (2022) proposed include angle information and build line graphs with complexity O(nk²) to increase the geometric modeling capacity, where n denotes the number of atoms in the unit cell and denotes the average number of
neighbors for every atom. Recently, Matformer (Yan et al. (0022) was proposed to encode periodic patterns by celf-coonn edges, and PotNet (Lin et al. (003)) considers the infinite summations of pairwise atomic interactions with additional computational cost O( Despite the successes of previous crystal neural networks, it remains unsolved to achieve geometric completeness for crystalline materials, with demonstrations provided in Appendix [A.]
Geometric completeness for molecules. To pursue geometric completeness for molecules, various approaches have been developed. SphereNet (Liu al., 2022) achieves geometric completeness for molecules in most cases by using the combination of Euclidean distances, bond angles, and torsional angles, with some failure cases. GemNet (Kliepera al. 2021) goes step further by incorporating dihedral angles and achieving completeness for molecules with complexity of ((nk"). ComENet (Wang et al., 2022) decreases the modcling complexity by selecting reference nodes using Euclidean distances. However, these methods fall short to capture periodic patterns (Yan et al.) 2022) of crystals and are not geometrically complete for crystalline materials. As another branch of work, Graphormer (Ying et al. 2021) uses fully-connected molecule graphs and achieves completeness for molecules, but breaks periodic invariance as shown Appendix 1.2
Complete representations for crystals. Constructing complete crystal representations in other forms is not a new topic. There are several recent efforts, including AMD (Widdowson et al. 2022) and PDD Widdowssn & Kurlin 2022) of matrix forms that are complete. However, AMD and PDD cannot distinguish chiral crystals with different properties. Additionally, using these matrix form representations input to predict crystal properties without breaking the completeness is challenging in practice. First, AMD and PDD representations are designed for stable crystal structures and do not consider atom types. The completeness of AMD and PDD is based on the assumption that no two crystals can have the same structure but differ by a single atom type, which is only feasible for stable crystal structures. Second, to achieve completeness, a sufficiently large number of neighbors needs to be predetermined for any test crystal (must be larger than the number of atoms in the cell for any test crystal), which not feasible and very expensive as mentioned in PDD-MI. (Balasingham et al.. 2022), with more details provided in Appendix A.133] It is worth noting that our proposed graph representations can apply to dynamic crystal systems and are robust to various cell sizes.
3 GEOMETRICALLY COMPLETE CRYSTAL GRAPH REPRESENTATIONS
In this section, we first demonstrate crystal passive symmetries that will not change crystal structures, including unit cell SE(3) invariance, unit cell SO(3) equivariance, and periodic invariance. We discuss consequences when these symmetries are not encoded. We then propose SE(3) invariant and SO(3) cquivariant crystal graph representations that achieve geometric completeness for crystalline materials. We further provide proofs that the proposed crystal graphs satisfy crystal passive symmetries in Appendix A.2 and are geometrically complete in Sec. 3.4 with verification provided in Appendix A.4
3.1 CRYSTAL PASSIVE SYMMETRIES
Crystal structures are different from molecules and proteins and exhibit unique passive symme- tries (Villar et al. 2023), including unit cell SE(3) invariance, unit cell SO(3) equivariance, and periodic invariance. Specifically, crystal structures remain unchanged when the position matrix P of the unit cell structure translated alone or rotated together with L, as shown in Fig. [b) and (c). Additionally, different minimum repeatable structures can be used to represent the same crystal as the periodic transformations shown in Fig. [d). and these different crystal structure representations (A, P. L) introduce crystal passive symmetry called periodic invariance. However, reflection
3

---

Published as a conference paper at ICLR 2024
(a) Original
(b) Translation
(c) Rotation
(d) Periodic Transtormations
(6) Reflection
Figure 1: Illustrations of crystal passive symmetries. We show examples of a real crystal structure in 3D and corresponding simpler demonstrations in 2D. For the 2D demonstrations, we circles with different colors to represent different kinds of atoms, and use red dotted lines to represent periodic unit cells. We show two cases of periodic transformations, including shifting the periodic unit cells and changing periodic unit cells without changing the volume. Translation, rotation, and periodic transformations will not change the crystal structure and are passive symmetries, while reflection transformation will map the crystal structure to its chiral image if the reflection symmetry is absent.
transformation is not crystal passive symmetry as it maps a crystal structure its chiral image. We then provide formal definitions of these crystal passive symmetries as follows.
Definition 2 (Unit Cell SE(3) Invariance). A function f (A, P. L) - X is unit cell SE(3) invariant for any rotation transformation R 3x3 R and translation transformation b R we have [(A, P. L) J(A, RP + b, RL).
A similar definition of unit cell E(3) invariance has been given in Yan et al. (002).. The major difference between unit cell F(3) and SE(3) invariance is that unit cell SE(3) invariant functions further distinguish crystal structures with different chiralities.
Definition 3 (Unit Cell SO(3) Equivariance). A function f: (A.P,I) - X' E R is unit cell SO(3) equivariant if. for any rotation transformation R E g3x3 6 E 3
R and translation transformation
we have (A L) = f(A, RP + b, RL).
For a given infinite crystal structure (P,A) where P = (p.pp = p: + k, + kzl2
kgk3, k1, k2, K3 E Σ, i E Z, 1 < n and A = {á; a; = a, Z, 1 < < n). periodic invariance describes that the representation of crystals should be invariant the choice of minimum unit cell representations. The definition is detailed below.
Definition 4 (Periodic Invariance). A function f (.P.PL) L)) - X is periodic invariant if. for any possible minimum unit cell representations M' (A',P', L') representing a given infinite crystal structure (P, we have f(A, P,L. f(A', P', L').
Two periodic transformations that can generate different minimum unit cell representations for the same crystal structure shown in Fig. dd), including shifting periodic boundaries shown in the left and changing periodic patterns with the same unit cell volume shown the right.
3.2 SE(3) INVARIANT CRYSTAL GRAPH REPRESENTATIONS
To achieve geometric completeness for crystalline materials, we introduce three periodic vectors, eli Gigg, and elig, to serve as the lattice representation node i. The lattice representations for different nodes are perfectly aligned because all atoms in crystal share the same periodic patterns.
Invariant crystal graph construction. Every node in the proposed SE(3) invariant crystal graph represents atom and all its infinite duplicates in 3D space, with positions (p.1p: P; k kele + kyLy, k1, 22, kis E Z} and node feature a, An edge will be built from node j to node i when the Fuclidean distance ||eyıll2 between duplicate of j and i satisfies ||eyılle = lp, + kie ki kkg Pill2 < r. where R is the cutoff radius determined by the k-th nearest neighbor. Each edge has a corresponding feature vector { {|[yrill By 1y....1....0 Ay'i, fy'i, }. with 8y 8yr, and fyrr,sk, denoting the angles between eys and lattice representation {ei, Gitz City J. respectively. Duplicates i, in, and in are included as neighbors of the lattice representation (ei, Giig Gigg } for node i can be found below.
to encode periodic patterns. The construction
4

---

Published as a conference paper at ICLR 2024
Importance of periodic invariance for lattice representation. A straightforward way to obtain lattice representation is {Ci, City, Clig = {li, L2, Ez}. where L1 l2, Lg are the periodic lattice vectors provided in input lattice matrix L. However,
shown in Sec. 3.1 and Fig. [ the same crystal structure can have different lattice matrices L, when applying periodic transformations. For instance, I. = 1 l2, fa)² and L' = [1 + f2. L2, lz)" describe the same periodic patterns and have the same volume of the unit cell. Therefore, lattice representation needs to be invariant to periodic transformations.
Lattice representation construction. The lattice representation is constructed as follows to satisfy periodic invariance. First, we select the periodic duplicate i of node that has the smallest edge length |eu,, ll2 S ||ewr 12 for any other duplicate i' where {p- = + k + kgl2 + kzl3, k1, k2. K3 E }}. We then select the periodic duplicate i2 of node that has the smallest edge length | ll2 S |ew for any other duplicate i' except i and verify that eū, and eii, are not in the same line. If they are in the same line in 3D space, we select the duplicate with the next smallest distance and repeat until eit, is found. We then adjust the vector eit, to -li, if the relative angle between eli, and eii, is larger than 90" Third, we select the duplicate with the next smallest distance ||eti and verify that City, Cit and Cúi, not in the same plane in 3D space, and repeat until Ciis is found, then adjust the vector Cii, to -i, if the relative angle between Ciiy and Cii, is larger than 90º. In the end, we will verify that (en, eii eli forms right-hand system, if not, convert them by (-eii Blig -eiig! } to form a right-hand system. The proposed lattice representation (eii Gitg Gigg] satisfies the unit cell SO(3) equivariance and periodic invariance, with detailed proofs provided in Appendix 19.221ix[A.2.1 We also provide proofs in Appendix I that it distinguishes different chiral crystal structures.
3.3 SO(3) EOUIVARIANT CRYSTAL GRAPH REPRESENTATIONS
In addition to SE(3) invariant crystal graphs, we propose SO(3) equivariant crystal graphs that achieve geometric completeness by using vector representations.
Equivariant crystal graph construction. The only difference between the proposed equivariant crystal graphs and invariant crystal graphs is the edge feature. Every edge in the equivariant crystal graph has the invariant edge feature ||eyil|2 and the equivariant edge feature eyi E R.3.
With 3D vectors as edge features, the unit cell SO(3) equivariant property can be achieved as proven in Appendix [A.2. However, SO(3) equivariant crystal graphs pose challenges to consider expressive geometric information during message passing. only the norm of 3D vectors were used during message passing, it will be the same previous methods using Eucldiean distances only, such as SchNet (Schütt et al., (017) and Matformer (aan et al., 2022).
3.4 GEOMETRIC COMPLETENESS OF PROPOSED CRYSTAL GRAPHS
Proposition 1. The SE(3) invariant and SO(3) equivariant crystal graphs are geometrically complete.
Proof. We prove by mathematical induction. We assume that the number of atoms (nodes) in the crystal unit cell is n. Note that the crystal graph we consider is strongly connected, which means that there exists a path between any two nodes in the graph. All crystalline materials in nature can be constructed as strongly connected graphs.
Base case: The infinite crystal structure represented by SE(3) invariant or SO(3) equivariant crystal graphs is unique when n = 1.
Inductive hypothesis: The infinite crystal structure is unique holds for n up to m 2
Inductive step: Let = m +1. 1. Without loss of generality, among the existing m nodes, we safely assume N₁ the set of nodes that form the local region of node j. Then j is the index of the (m + 1)-th node newly connected to these nodes. To prove the infinite crystal structure is still unique, we only need to prove that the relative position of node j is uniquely determined given the SE(3) invariant crystal graph or the SO(3) equivariant crystal graph. The proofs of Base and Inductive cases are provided in Appendix A.3 With that, the proposed SE(3) invariant and SO(3) equivariant crystal graphs can determine unique infinite crystal structure. Hence, the same crystal graphs proposed will only represent the identical infinite crystal structure. Then, based on Definition I we complete the proof of Proposition 1
5

---

Published as a conference paper at ICLR 2024
Passive Symmetries
SEX) Crystal Graph
ComForeer
Oe: dy |equil Byn..
embedding
og
node-wise transtormer
Equivariant
egi
||opl %
edge-wise
||sl||e
equivarian:
upataal
Figure 2: Illustration of the proposed ComFormer pipeline. In the left figure, show different unit cell structures for the same crystal due to passive crystal symmetries, and all of them will map to the same invariant equivariant crystal graph shown in the middle. In the middle, information included in our proposed SE(3) invariant and equivariant crystal graphs. Specifically, we include node feature a, for every node and for every neighbor j of node i, include edge length |eeji l2. and three angles Oji = {O Ojt.sig; Ofi,ti } in invariant crystal graphs, and edge vector ej in equivariant crystal graphs. The proposed iComFormer and eComFormer are shown on the right, with building blocks marked by different colors.
demonstrate the
4 EFFICIENT AND EXPRESSIVE CRYSTAL TRANSFORMERS
We then propose iComFormer and eComFormer for crystal representation learning, based on the proposed SE(3) invariant and SO(3) equivariant crystal graphs in efficient manner.
4.1 SE(3) INVARIANT MESSAGE PASSING
In this section, we first present iComFormer as shown the upper right panel of Fig. iComFormer utilizes SE(3) invariant crystal graphs with computational complexity of O(nk), where n is the number of nodes, and k is the average number of neighbors for every node.
Node and edge feature embedding layer. Node feature ag for node is mapped to f using CGCNN (Xie & Grossman 2018) embedding features as the initial node feature before transformer layers. Edge feature |/eull2 for node pair j and is mapped to c/ e/[eeill2 and embedded by RBF kernels to initial edge feature f; where c is chosen constant to mimic the pairwise potential calculation following Lin et al. 2023). Three angles 85,, ayg, and fya,ts are mapped fj f/12 and jig by cosine function and RBF kernels. More details are provided in Appendix A.6
Node-wise transformer layer. Node-wise transformer layer first passes messages from a neighboring node j to the center node using node features f/. ft and edge feature fū, where indicates the layer number, and then aggregates all neighboring messages to update ! The message from node to i is formed by the corresponding query 4is, key kjj, and value features Uys as follows,
qj = L k;; = x (f)) (z) By = (LNv (
ajji 9jz o O (kec) ddaaa
msgj = (BN(a,d(BN(a,.))) oov(U,)
(2)
where LNQ. LNK, LNv, L. h are the linear transformations for query, key, value, and edge features. ØK σy are the nonlinear transformations for key and value, including two linear layers and an activation layer in between. and denote the Hadamard product and concatenation, respectively. BN denotes the batch normalization layer, and dq,, is the dimension of qıi. In design, the query in the transformer layer only contains the center node feature, while the key and value contain the center node, neighboring node, and edge features. Then, node feature f is updated using aggregated messages from neighbors to center node as follows,
msgi Σ msgji,
f!+1 (f BN(msg))).
JEM.
where Omag denotes the softplus activation function. Because we calculate messages for every node i where h denotes the average degree, the complexity of node-wise transformer layer O(nk).

---

Published as a conference paper at ICLR 2024
Edge-wise transformer layer. Edge-wise transformer layer updates edge feature ff using angle features f f/.y and fig. and edge features of lattice vectors f figz and fāg Detailed updating computation of edge-wise transformer layer is as follows,
9; -LN fin. = ( (FJJILN (f::... 1), Wiim = (LN (FJOILL (R..))
Li: ook (k.... (S msgj... = Higmoi((BN(a)))) o [LNo(S)) )),
a fm
daa.
(4)
where LN LN L represent linear transformations for edge feature fj. LN L. m E {1, 2,33 represent linear transformations for edge features of lattice vectors, and LN, is the linear transformation for angle features f f9 fts- Then, edge feature fj: is updated using aggregated messages from three lattice vectors as below.
msgji
Σ m0
msgj....
fR msg BN(msg
(5)
because every edge ji has three neighbors, the complexity of edge-wise transformer layer ((3n)) O(nk). Hence, the total complexity of iComFormer is O(nk). The proof that iCom- Former considers expressive geometric information in message passing is shown in Appendix
4.2 SO(3) EQUIVARIANT MESSAGE PASSING
In this section, we describe SO(3) equivariant message passing for eComFormer, as shown in the lower right panel of Fig. 2 For the design of eComFormer message passing, core question answer how to capture expressive geometric information in SO(3) crystal graphs. Specifically, different from Euclidean distances, vector features eji do not support nonlinear mappings. To tackle this, employ the recent advances in tensor product (TP) layers (Geiger & Smidi, 2022a) and capture the expressive geometric information by properly stacking several TP layers together.
Node and edge feature embedding layer. Node feature a, and edge feature ||eull2 for node pair and are embedded in the way iComFormer. Equivariant edge feature Gji is embedded by corresponding spherical harmonics Yu Yu(ej) co. Ye(ej)) C1 eja E RR3 and Y2(ej)) E R where
ej and Co2 C1 are constants, with more details provided in Appendix A.6 Teejilla
eji
Node-wise equivariant updating layer. To capture geometric information represented by vector features eji in SO(3) crystal graphs, we propose the node-wise equivariant updating layer. Specifically, implicitly considers expressive geometric information by stacking two tensor product layers and taking edge features (Yo(ejı), (eje),, Y2 (eju)) and node features f as input. The first tensor product layer aggregates higher rotation order neighborhood information the center node as follows,
Σ TPo(F). Yo(o,)). JEN,
Σ TPX(F Yx(e))), X (1,2), (6) JEN,
W
where ff = LN(F)). W is the number of neighbors of node i, and TPp, TP1, TP2 are tensor product layers that produce outputs with rotation order equal 0, 1, and 2, respectively. Then, the second tensor product layer produce SE(3) invariant node features as below,
fl
TPo Yo(eja))
Σ
TPo(F) Y (e(e))
WI
JEN
JENS
JEN.
flupdettdd Equul (N)( + (Ne)
where requi is a nonlinear transformation consisting of two softplus layers and a linear layer in between. We then show that by the proper design of tensor product layers, the output Plupdatedd of node-wise equivariant updating layer considers all two-hop bond angle information EjeN, EmEN, COS(Omgai) in Ejen, TPo(f Y, (eja)), with details provided in Appendix A.5
7

---

Published as a conference paper at ICLR 2024
5 EXPERIMENTS
We assess the expressiveness of our iComFormer and eComFormer models by conducting evalu- ations on three widely-used crystal benchmarks: JARVIS (Choudhary et al. 2020), the Materials Project (Chen et al., 2019), and MatBench (Dunn et al. 2020). Through extensive experiments, proposed ComFormer exhibits excellent prediction aceuracy across a range of crystal properties, underscoring its modeling capabilities for crystalline materials.
For JARVIS and MP, we follow the experimental settings of Matformer Yan et al. 2022) and PotNet (Lin al., 2023), and use mean absolute error (MAE) the evaluation metric. Among these tasks, there are large scale tasks with 69239 crystals, medium scale tasks with 18171 crystals, and small scale tasks with 5450 crystals. To further evaluate the performances, we use e_form with 132752 crystals and jdft2d with only 636 2D crystals in MatBench. The baseline methods include CGCNN (Xie & Grossman 2018), MEGNET (Chen et al. 2019), GATGNN (Louis al. 2020). ALIGNN (Choudhary & DeCost 2021), Matformer ( (Yan et al. 2022). PotNet (Lin et al. 2023). M3GNet (Chen Ong. 2022), MODNet (De Breuck et al. 2021), and coGN (Ruff et al. 2023) For tasks in these three datasets, the best results are shown in bold and the second best results are shown with underlines. We use one TITAN A100 for computing. More details ComFormer settings and benchmark details can be found in Appendix and
and A.7 respectively.
5.1 EXPERIMENTAL RESULTS
JARVIS. As shown in Table I com- pared with highly competitive base- lines, ComFormer variants achieve the best performances on all tasks. Specifically, iComFormer achieves 8% improvement beyond PotNet on formation energy, 10% improvement on Exotal, and 15% improvement Exat, while eComFormer achieves 20% improvement on Enalt and the second best performances on other three tasks.
Table 1: Comparison JARVIS in terms of test MAE,
Fonn. Energy
Bandgap(OPT)
Eist!!
Bandgap(MBB)
E
Method
MeNtatom
mc//stom
CGCNN
0.20
78
0.4]
170
SchNet
45
0.19
47
0.43
140
MEGNET GATGNN
0,145
58
0.34
0.170
ALIGNN
33.1
0.142
37
0.31
76
Matfermer
32.5
0.137
35
0.50
64
PoiNet
29.4
0.127
32
0.27
eComFormer
iComFormer
27.2
0.122
28.8
0.26
47
The Materials Project (MP). Experi- mental results on MP are shown in Ta- ble 2 It can be seen that iComFormer achieves the best performances on three out of four tasks with significant margins beyond previous works, while eCom- Former achieves the best formation energy. Specifically, the excellent pre- diction accuracy of iComFormer on bulk moduli and shear moduli with only 4664 training samples shows the expressive- ness and robustness of the invariant crystal graphs with limited training samples.
Table 2: Comparison on The Materials Project.
Form. Energy
Band Gup
Bulk Moduli
Shear Moduli
Method
ma Vatom
eV
logrGPa)
log(GPa)
CGCNN
0.292
0.047
0.077
SchNet
33
0.345
0.066
0.099
MEGNET
GATGNN
33
0.280
0.045
0.075
AI ALIGNN
0.218
0.051
0.078
Matformer
21.0
0.211
0.043
0.073
PotNet
18.8
0,204
0.040
0.0650
comFoommer
18.16
0.202
0.0417
0.0729
iComFormer
18.26
0.193
0,0380
0.0637
MatBench. Experimental results for MatBench are shown in Table terms of MAE and RMSE with standard deviation of five benchmark runs. For the largest scale e form with more than 130K crystals, ComFormer variants achieve the state-of-the-art per- formances, shedding light on the elegant integration of angle information in the complete crystal repre- sentations beyond other methods using angle infor- mation like ALIGNN and M3GNet. For jdft2d task, iComFormer is competitive with MODNet, the state-of-the-an physical descriptor based method for materials, indicating the excellent modeling power of iComFormer with only hundreds training samples. Overall, the robust performances of ComFormer variants these three benchmarks for
Table 3: Comparison on MatBench.
e_ferm 132752
38128-634 636
MODNEr
448
96.7
ALIGNN
21.5 1.0.
55.4 5,5
43.4
117.1 4299
COEN
1700 070300
5.9
57.2 13.7
101.2 55.0
MBNNe
19.5
50.1 0:1:::1 11.9
ComFormeer
165
45
57.81990
102.2 1: 46.4
Konnifommrr
16.5 2 0.3
43.8
3.
3488
1 9.
96.1 46.3
8

---

Published as a conference paper at ICLR 2024
various crystal properties with different data scales demonstrate the effectiveness of the proposed invariant and equivariant crystal representations and ComFormer architectures.
Efficiency. We evaluate the model complexity and running time of ComFormer variants by compar- ing them with geometrically incomplete methods Matformer and ALIGNN on the JARVIS formation energy task. To make the comparison fair, fol- low Matformer and ALIGNN and use k = 12 when constructing crystal graphs. Further analysis when increasing & is provided in Appendix A.6.2] We use and indicate the number node layers of iCome, which covers all experimental settings of iComFormer. As shown in Table our proposed ComFormer variants are significantly more efficient than ALIGNN and are comparable with Matformer which only uses Euclidean distances.
Table 4: Efficiency analysis.
Method
Complexity
Num. Params.
Time/epoch
Matformer
O(nk)
2,9 M
64
ALIGNN
O(nk9)
327
cComFormer
Ojak)
12.4
115
cComoormc-baaf
Oink)
M
90
iComFormen3)
O(nkj
4.1 M
:ComFormen4)
O(ak)
5.0
78
Table 5: Verification of geometric completeness of proposed SE(3) invariant and SO(3) equivariant crystal graphs by reconstructing crystal structures from graph representations.
Structure RMSD
MP
T2 Pulido et al. 2017)
SE(3) invariant
3.28 10-7
3.83 * 10-7
SO(3) equivariant
3.03 10-7
3.19 * 10-7
Completeness verification. We evaluate the completeness of proposed SE(3) invariant and SO(3) equivariant crystal graphs by reconstructing crystal structures from graph representations. Specifically, crystals from the Materials Project and T2 dataset which contains super large crystal cells with more than 700 hundred atoms to demonstrate the robustness of these crystal graphs. The detailed reconstruction process is provided in Appendix, A.44 After reconstruction, we compare the structure differences between the recovered ones and the input ones using Pymatgen. As shown in Table, 5 the input crystal structures are fully recovered, with limited structure error (less than 10-6).
5.2 ABLATION STUDIES
In this section, we demonstrate the importance of geometric complete crystal graphs and Comformer components by conducting ablation studies on JARVIS formation energy task.
Importance of geometric complete crystal graphs. We evaluate the importance of geometric complete- ness by making the proposed crystal graphs incomplete. Specifically, we omit the edge-wise transformer layer in iComFormer and the equivariant updating layer in eComFormer which means that we omit bond angles and bond vectors in corresponding graphs. As shown in Table 6 using the incomplete crystal graphs will degrade the performance of iComFormer by 6.25% from 0.0272 to 0.0289, and the performance of eComFormer by 6.33% from 0.0284 to 0.0302.
Table 6: Importance of completeness.
Method
Complete
Test MAE
w/o bond vectors
x
30.2
eComFormer
/
28.4
w/o bond angles
x
28.9
iComFormer
/
27.2
CONCLUSION, LIMITATIONS. AND FUTURE WORKS
In this work, we propose SE(3) ComFormer variants for crystalline materials that utilize geometrically complete crystal graphs and satisfy crystal passive symmetries. The state-of-the-ar performances of ComFormer variants on various tasks across three widely-used crystal benchmarks demonstrate the expressive power of proposed geometrically complete crystal graphs and modeling capacity of ComFormer variants. The limitations of our current method include (1) it is designed for crystalline materials and cannot be directly applied to other domains, (2) cannot model the complete geometric information of amorphous materials, and (3) it currently cannot predict higher order crystal properties. We plan to explore these directions in the future work. We also include discussions of potential corner cases in Appendix A.8 to aid future users of the proposed method in need.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
We thank Youzhi Luo, Limei Wang, and Haiyang Yu for insightful discussions, and thank Chengkai Liu for drawing 3D visualizations of crystal structures. S.J. acknowledges the support from National Science Foundation grants IIS-2243850 and CNS- CNS-2328395. XF Q. acknowledges the support from the Center for Reconfigurable Electronic Materials Inspired by Nonlinear Dynamics (reMIND), an Energy Frontier Research Center funded by the U.S. Department of Energy, Basic Energy Sciences, under Award Number DF-SC0023353. XN Q. acknowledges the support from National Science Foundation grants CCF- 155-155328,, DMR-21 19103, and IIS-2212419.
REFERENCES
Zeeshan Ahmad, Tian Xie, Chinmay Maheshwari, Jeffrey C Grossman, and Venkatasuuraamanian Viswanathan. Machine learning enabled computational screening of inorganic solid electrolytes for suppression of dendrite formation in lithium metal anodes. ACS central science, 4(8):996-1006. 2018.
Olga Anosova and Vitaliy Kurlin. Algorithms for continuous metrics on periodic crystals. arXiv preprint arXiv:2205. 15222 2022.
Jonathan Balasingham, Viktor Zamaraev, and Vitaliy Kurlin. Compact graph representation of molecular crystals using point-wise distance distributions. arXiv preprint arXiv:2212.11246 2022.
Simon Batzner, Albert Musaelian, Lixin Sun, Mario Geiger, Jonathan P Mailoa, Mordechai Kornbluth, Nicola Molinari, Tess E Smidt, and Boris Kozinsky. E(3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials. Nature communications, 13(1):2453, 2022.
Lowik Chanussot, Abhishek Das, Siddharth Goyal, Thibaut Lavril, Muhammed Shuaibi, Morgane Riviere, Kevin Tran, Javier DorinnDD Caleb Ho, Weihua Hu, et al. Open catalyst 2020
(oc20) dataset and community challenges. Acs Catalysis, 11(10):6059 0001:666996662...0 2021.
Chi Chen and Shyue Ping Ong. A universal graph deep learning interatomic potential for the periodic table. Nature Computational Science, 2(11):718-728 2022.
Chi Chen, Weike Ye, Yunxing Zuo, Chen Zheng, and Shyue Ping Ong. Graph Networks as a Universal Machine Learning Framework for Molecules and Crystals. Chemistry of Materials, 31 (9):3564-3572, 2019.
Kamal Choudhary and Brian DeCost. Atomistic Line Graph Neural Network for improved materials property predictions. npi Computational Materials, 7(1):1-8, 2021.
Kamal Choudhary, Kevin F Garrity, Andrew CE Reid, Brian DeCost, Adam Biacchi, Angela R Hight Walker, Zachary Trautt, Jason Hattrick-Simpers, A Gilad Kusne, Andrea Centrone, al. The joint automated repository for various integrated simulations (JARVIS) for data-driven materials design. npj Computational Materials, 6(1):1-13, 2020.
Pierre-Paul De Breuck, Geoffroy Hautier, and Gian-Marco Rignanese. Materials property pre- diction for limited datasets enabled by feature selection and joint learning with MODNet. npi Computational Materials, 7(1):83, 2021.
Alexander Dunn, Qi Wang. Alex Ganose, Daniel Dopp, and Anubhav Jain. Benchmarking materials property prediction methods: the matbench test set and automatminer reference algorithm. npj Computational Materials, 6(1): 2020.
Herbert Edelsbrunner, Teresa Heiss, Vitaliy Kurlin, Philip Smith, and Mathijs Wintraecken. The density fingerprint of periodic point set. arXiv preprint arXiv:2104.1 2021.
Cong Fu, Xuan Zhang, Huixin Zhang, Hongyi Ling, Shenglong Xu, and Shuiwang Ji. Lattice convolutional networks for learning ground states of quantum many-body systems. arXiv preprint arXiv:2206.07:300 2022.
Cong Fu, Keqiang Yan, Limei Wang. Wing Yee Au, Michael McThrow, Tao Komikado. Koji Maruhashi, Kanji Uchino, Xiaoning Qian, and Shuiwang Ji. A latent diffusion model for protein structure generation. arXiv preprint arXiv:2305.0411 2023.
10

---

Published as a conference paper at ICLR 2024
Hongyang Gao and Shuiwang Ji. Graph U-Nets. In International Conference on Machine Learning, pp. 209822002.. PMLR, 2019.
Johannes Gasteiger, Janek GroB, and Stephan Günnemann. Directional message passing for molecular graphs. arXiv preprint arXiv:2003.03122 2020.
Mario Geiger and Tess Smidt. e3nn: Euclidean Neural Networks. arXiv preprint arXiv:2207.09453, 2022a.
Mario Geiger and Tess Smidt, e3nn: Euclidean neural networks. arXiv preprint arXiv:2207.09453. 2022b.
Anubhav Jain, Shyue Ping Ong, Geoffroy Hautier, Wei Chen, William Davidson Richards, Stephen Dacek, Shreyas Cholia, Dan Gunter, David Skinner, Gerbrand Ceder, et al. Commentary: The materials project: A materials genome approach to accelerating materials innovation. APL. materials, 1(1):01 1000010022)) 1002, 2013.
Diederik P Kingma and Jimmy Ba. Adam: Method for Stochastic Optimization. In Proceedings of the 3rd International Conference on Learning Representations, 2015.
Johannes Klicpera, Florian Becker, and Stephan Günnemann. GemNet: Universal Directional Graph
Neural Networks for Molecules. In Advances Neural Information Processing Systems, 2021.
Yi-Lun Liao and Tess Smidt. Equiformer: Equivariant graph attention transformer for 3d atomistic graphs. arXiv preprint tcXiv:2206.1199) 2022.
Yuchao Lin, Keqiang Yan, Youzhi Luo, Yi Liu, Xiaoning Qian, and Shuiwang Ji. Efficient ap- proximations of complete interatomic potentials for crystal property prediction. In ICML, pp. 21260-21287 2023.
Hongyi Ling, Zhimeng Jiang, Youzhi Luo, Shuiwang Ji, and Na Zou. Learning fair graph representa- tions via automated data augmentations. In The Eleventh International Conference on Learning Representations, 2023. /ttt/httt////eeetti
net. wtt.//orrmmuoooo// wtwp:mmiiiii_11.__11_11 -t____GWPP_ee 9w
Meng Liu, Cong Fu, Xuan Zhang, Limei Wang, Yaochen Xie, Hao Yuan, Youzhi Luo, Zhao Xu, Shenglong Xu, and Shuiwang Ji. Fast quantum property prediction via deeper 2d and 3d graph networks. arXiv preprint arXiv:2106 106.08551, 2021a.
Meng Liu, Youzhi Luo, Limei Wang, Yaochen Xie, Hao Yuan, Shurui Gui, Haiyang Yu, Zhao Xu, Jingtun Zhang, Yi Liu, et al. DIG: turnkey library for diving into graph deep learning research. Journal of Machine Learning Research, 22(240):1-9, 2021b.
Meng Liu, Keqiang Yan, Bora Oztekin, and Shuiwang Ji. GraphEBM: Molecular graph generation with energy-based models. arXiv preprint arX0v:2102.00546, 2021c
Yi Liu, Limei Wang, Meng Liu, Yuchao Lin, Xuan Zhang, Bora Oztekin, and Shuiwang Ji. Spherical message passing for 3D molecular graphs. In International Conference on Learning Representations, 2022.
Steph-Yvee Yves Louis, Yong Zhao, Alireza Nasiri, Xiran Wang. Yugi Song, Fei Liu, and Jianjun Hu. Graph convolutional neural networks with global attention for improved materials property prediction. Physical Chemistry Chemical Physics, 12(32):11441-1814 2020.
Youzhi Luo, Keqiang Yan, and Shuiwang Ji. GraphDF: A Discrete Flow Model for Molecular Graph Generation. In International Conference on Machine Learning, pp. 7192-7203. PMLR, 2021.
Bryce Meredig, Ankit Agrawal, Scott Kirklin, James E Saal, Jeff W Doak, Alan Thompson, Kunpeng Zhang, Alok Choudhary, and Christopher Wolverton. Combinatorial screening for new materials in unconstrained composition space with machine learning. Physical Review B, 8999)4444 2014.
Anton O Oliynyk, Erin Antono, Taylor D Sparks, Leila Ghadbeigi, Michael W Gaultois, Bryce Meredig, and Arthur Mar. High-throughput machine- seatee--aannnn--iiiee synthesis of full-heusler compounds. Chemistry of Materials, 28(20):724-733) 2016.

---

Published as a conference paper at ICLR 2024
Saro Passaro and C Lawrence Zitmick. Reducing SO(3) convolutions SO(2) for efficient equivariant GNNs. arXiv preprint arXiv:2302.03655, 2023.
Angeles Pulido, Linjiang Chen, Tomasz Kaczorowski, Daniel Holden, Marc A Little, Samantha Y Chong, Benjamin Slater, David P McMahon, Baltasar Bonillo, Chloe J Stackhouse, et al. Funetional materials discovery using energy structure-functene maps. Nature, 543(7647):657- 664, 2017.
Zhuoran Qiao, Anders S Christensen, Matthew Welborn, Frederick R Manby, Anima Anandkumar, and Thomas F Miller III. Informing geometric deep learning with electronic interactions to acceler- quantum chemistry. Proceedings of the National Academy of Sciences, 119(31 ):e20022211 2022.
Paul Raccuglia, Katherine C Elbert, Philip DF Adler, Casey Falk, Malia B Wenny, Aurelio Mollo, Matthias Zeller, Sorelle A Friedler, Joshua Schrier, and Alexander J Norquist. Machine-learning assisted materials discovery using failed experiments. Nature, 533(7601):73-777 2016.
Rampi Ramprasad, Rohit Batra, Ghanshyam Pilania, Arun
and Chiho Kim. Machine learning in materials informatics: recent applications and prospects. npi Computational Materials, 3(1):1-13, 2017.
Jakob Ropers, Marco M Mosca, Olga Anosova, Vitaliy Kurlin, and Andrew I Cooper. Fast predictions of lattice energies by continuous isometry invariants crystal structures. In International Conference on Data Analytics and Management in Data Intensive Domains, pp. 178-192. Springer, 2021.
Robin Ruff, Patrick Reiser, Jan Stühmer, and Pascal Friederich. Connectivity optimized nested graph networks for crystal structures, arXiv preprint arXiv:2302.14102 2023.
Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E(n) equivariant graph neural networks. In International conference on machine learning, pp. 9323-9332. PMLR, 2021.
Kristof Schütt, Pieter-Jan Kindermans, Huziel Enoc Sauceda Felix, Stefan Chmiela, Alexandre Tkatchenko, and Klaus-Robert Müller. SchNet: Continuous- Convolutional Neural Network for Modeling Quantum Interactions. Advances in Neural Information Processing Systems, 30, 2017.
Leslie N Smith and Nicholay Topin. Super-convergence: Very fast training of residual networks using large learning rates. 2018.
Soledad Villar, David W Hogg, Weichi Yao, George A Kevrekidis, and Bernhard Schölkopf. The passive symmetries of machine learning. arXiv preprint arXiv:2301.13724 2023.
Limei Wang, Haoran Liu, Yi Liu, Jenry Kurtin, and Shuiwang Ji. Learning hierarchical protein representations via complete 3d graph networks. In The Eleventh International Conference on Learning Representations.
Limei Wang, Yi Liu, Yuchao Lin, Haoran Liu, and Shuiwang Ji. ComENet: Towards complete and efficient message passing for 3D molecular graphs. In The 36th Annual Conference on Neural Information Processing Systems, pp. 650-664, 2022.
Logan Ward, Ankit Agrawal, Alok Choudhary, and Christopher Wolverton. A general- machine learning framework for predicting properties of inorganic materials. npi Computational Materials, 2(1):117, 2016.
Joseph L Watson, David Juergens, Nathaniel R Bennett, Brian L Trippe, Jason Yim, Helen E Eisenach, Woody Ahem, Andrew Borst, Robert J Ragotte, Lukas F Milles, et al. Broadly applicable and accurate protein design by integrating structure prediction networks and diffusion generative models. bioRxiy, pp. 2022-12, 2022.
Daniel Widdowson and Vitaliy Kurlin. Resolving the data ambiguity for periodic crystals. Advances in Neural Information Processing Systems, 35:24625-2444 -4668,, 2022.
12

---

Published as a conference paper at ICLR 2024
Daniel Widdowson, Marco M Mosca, Angeles Pulido, Andrew I Cooper, and Vitaliy Kurlin. Average minimum distances of periodic point fets-foundttional invariants for mapping periodic crystals. CIUCH-COMMUNICATTONN 87(3):529-55 2022.
IN MATHEMATICAL AND IN COMPUTER CHEMISTRY,
Tian Xie and Jeffrey C Grossman. Crystal Graph Convolutional Neural Networks for Accurate and Interpretable Prediction of Material Properties. Physical Review Letters, 120(14):145301 2018.
Tian Xie, Xiang Fu, Octavian-Eugen Ganea, Regina Barzilay, and Tommi Jaakkola. Crystal Diffu- sion Variational Autoencoder for Periodic Material Generation. In International Conference on Learning Representations, 2022.
Kegiang Yan, Yi Liu, Yuchao Lin, and Shuiwang Ji. Periodic graph transformers for crystal material property prediction. In The 36th Annual Conference on Neural Information Processing Systems, 2022.
Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie-Yan Liu. Do Transformers Really Perform Badly for Graph Representation? Advances in Neural Information Processing Systems, 34, 2021.
Haiyang Yu, Limei Wang, Bokun Wang, Meng Liu, Tianbao Yang, and Shuiwang Ji. GraphFM: Improving large-scale GNN training via feature momentum. In International Conference on Machine Learning, pp. 35684-25700 PMLR, 2022.
Xuan Zhang. Limei Wang, Jacob Helwig, Youzhi Luo, Cong Fu, Yaochen Xie, and Shuiwang Ji. Artificial intelligence for science in quantum, atomistic, and continuum systems. ar preprint arXiv:2307.08423, 2023.
A APPENDIX
A.1 LIMITATIONS OF PREVIOUS METHODS IN ACHIEVING GEOMETRIC COMPLETENESS FOR CRYSTALLINE MATERIALS
A.1.1 LIMITATIONS PREVIOUS CRYSTAL GRAPH NEURAL NETWORKS
Crystalline materials are first converted into graph representations which serve as the input of crystal graph neural networks. Thus, the geometric information considered by the crystal graph neural networks are directly determined by the employed crystal graphs.
In this subsection, we discuss the limitations of previous crystal graph neural networks in terms of crystal graph representations. These methods include CGCNN (Xie & Grossman 2018).
SchNet (Schütt et al. (017)). MEGNET (Chen et al. (2019), and GATGNN (Louis et al. 2020) which directly use multi-edge crystal graphs, ALIGNN (Choudhary & DeCost 2021) and M3GNet (Chen
Ong 2022) which uses multi- eegge crystal graphs with extra bond angles, and Matformer (Yan et al. 2022
and PotNet (Lin et al. 2023) which use additional edge features to capture periodic patterns.
We first recap multi-edge crystal graphs, and then show the limitations of previous works in distin- guishing different crystalline materials.
Multi-edge crystal graphs. The multi-edge crystal graph is proposed by Xie & Grossman (2018) to capture atomic interactions across periodic boundaries. As shown Fig. 3 every node in the multi-edge crystal graph represents atom and all its infinite duplicates in 3D space, with positions
{p. p. p: + ki é + k2l2 + kyls, ki, h2,, k3 E 2} and node feature a; An edge will be built from node node when the Euclidean distance |/erilla between duplicate of j and satisfies
leyill2 = Ip, + ki + k, + Kgla Pilla S r, where r E R is the cutoff radius. Each edge has a corresponding edge feature ||cy:ll2.
Limitations of multi-edge crystal graphs. (1) It can be seen from Fig, 4 that multi-edge crystal graphs fail short to distinguish erystals with different periodic patterns. and may map different crystals with different properties to the same graph. (2) It can be seen from Fig. [5 that multi-edge crystal graphs fail short to distinguish crystalline materials with different unit cell structures,
13

---

Published as a conference paper at ICLR 2024
y
N
lz)
e1
(a)
(c)
Figure 3: Demonstration of the multi-edge crystal graph. We use blue arrows to represent periodic patterns & and l2 for the shown crystal structure in 2D case. We use circles with different colors to represent different atoms, and light-green lines to represent the periodic boundaries. We use N; to represent the neighborhood of node within the radius, and use red and green arrows to indicate the captured atomic interactions by the multi-edge crystal graph. (a) A crystal structure with periodic patterns & and & in 2D case for clarity. (b) The captured atomic interactions by the multi-edge crystal graph, between the red nodes j and center node i. (c) The corresponding multi-edge crystal graph. All periodic duplicates of j mapped single node j in the multi-edee eddee crystal graph.
81
Figure 4: Demonstration that multi-edge crystal graphs fail short to capture periodic patterns. We use blue arrows to represent periodic patterns & and &g for these two different crystal structures. Due to the missing geometric information of periodic patterns, multi-edge crystal graphs map these two different crystals to the same crystal graph G.
and may map different crystals with different properties to the same graph. Thus, CGCNN (Xie & Grossman 2018), SchNet (Schütt et al. (017). MEGNET (Chen et al. 2019), and GATGNN (Louis al. 2020) which directly multi-edge crystal graphs fail short achieve geometric completeness
for crystalline materials.
Limitations of encoding bond angles using multi-edge crystal graphs. Noting that further encoding bond angles using line graphs upon multi-edge crystal graphs still fail short to capture periodic patterns, as shown in Fig. Additionally, further encoding bond angles using line graphs upon multi-edge crystal graphs still cannot distinguish crystals shown in Fig. since bond angles are identical for these two crystals. Hence, ALIGNN (Choudhary DeCost 2021) and M3GNet (Chen & Ong. 2022) which uses multi-edge crystal graphs with extra bond angles may map crystals with different properties to the same crystal graphs.
Limitations of Matformer and PotNet. By directly encoding periodic patterns using self-connecting edges, Matformer (Yan et al., 2022) can distinguish the two crystals shown Fig. 4 However, Matformer cannot distinguish crystals with the same periodic patterns but different unit cell structures as shown in Fig. 5 due to the fact that the Euclidean distances encoded in the crystal graphs are identical. PotNet (Lin et al. 2023) encodes periodic patterns by using infinite summations of two- body atomic potentials. However, PotNet only considers two-body interactions and using summation operation with errors, resulting in the inability to achieve geometric completeness.
14

---

Published as a conference paper at ICLR 2024
and ungle features
edge and angle fealurea
jack
Figure 5: Demonstration that multi-edge crystal graphs fail short to distinguish crystalline materials with different unit cell structures. We use i, k to represent three atoms the unit cell structures of these two crystals, d1, d2 to represent the encoded bond lengths in the graph, and jju to represent the encoded bond angle between edge ji and These two crystals are different but have the identical multi-edge crystal graphs with identical Euclidean distances and bond angles encoded. The number of blue atoms in the middle part is three, and can increase to any arbitrary number to obtain the same conclusion.
Figure 6: Demonstration that directly using molecule graphs will break periodic invariance. We use green circles denote the local radius region for every atom in the unit cell. The two crystals shown above are identical and obtained by using periodic transformation (shifting the periodic boundaries). Molecule graphs treat every atom within the radius single node in the constructed graph, and will include two blue nodes in total for the left crystal but include three blue nodes in total for the right crystal. This because molecule graphs break periodic invariance for crystals.
A. A.1.2 LIMITATIONS OF PREVIOUS COMPLETE MOLECULE GRAPH NEURAL NETWORKS
To achieve completeness for molecules with finite number of atoms, the molecule graphs used radius graphs and fully-comnected graphs.
Limitations of methods using radius graphs. SphereNet (Liu et al. 2022 and ComENet (Wang et al. 2022) use radius graphs, which fail short to capture periodic patterns similar to the multi-edge crystal graphs as shown Fig. 4 Additionally, molecule graphs break periodic invariance for crystal structures and will generate graphs with different number of nodes for the same crystal as shown in Fig. 6
I Limitations of methods using fully-connected graphs. Another approach to achieve geometric com- pleteness for molecules will be constructing fully-connected graphs. However, erystalline materials are periodic structures, which makes constructing colleccondeeted graphs infeasible. Additionally, constructing fully-connected graphs for super cell structures will break periodic invariance as shown in Fig.
A.1.3 LIMITATIONS OF PREVIOUS COMPLETE CRYSTAL MATRIX FORM REPRESENTATIONS
There recent works, AMD-ML (Ropers al. 2021) and PDD-ML Balasingham al. 2022 that use previously established powerful matrix form crystal representations, AMD and PDD.
Limitations of AMD-ML. Although AMD-ML uses complete crystal matrix representation AMD, it converts the matrix form representation to a single invariant vector using summation as the input of machine learning algorithms. By doing this transformation, the vector itself cannot fully recover the
15

---

Published as a conference paper at ICLR 2024
Figure 7: Demonstration that constructing fully-connected graphs for super cell crystal (expanding unit cell structures using surrounding unit cells) structures breaks periodic invariance (cannot remain the same when the periodic boundaries are shifted).
crystal structure and loses some structural information. Additionally, AMD-ML does not consider atom-type information of crystals and can only be used for stable crystal structures.
Limitations of PDD-ML. PDD-ML more powerful than AMD-ML by converting the PDD matrix into crystal graphs like CGCNN- -gaph.. By doing so, PDD-ML considers atom-type information. However, to achieve completeness for the PDD matrix, a large enough & needs to be predetermined and fixed before training for any future test samples, where k needs to be larger than the maximum number of atoms in the cell for all test crystals. This means that the PDD-MI. graph needs to have larger than 200 for The Materials Project, and larger than 700 for the T2 dataset that they tested on. Due to this practical concern, PDD-ML only uses kmaz = 25, which is far from completeness for their experiments on the Materials Project and T2 dataset. Additionally, the geometric information considered in PDD-ML for a fixed number of neighbors is the same as CGCNN with the same k as mentioned by the authors and verified by experiments in their paper. As demonstrated by extensive experimental results, the proposed Comformer is significantly more effective than CGCNN in all tasks. Additionally, by using k = 25 nearest neighbors, our proposed graph representations achieve completeness for the T2 dataset shown in Tab. g
A.2 PROOFS THAT THE PROPOSED CRYSTAL GRAPH REPRESENTATIONS AND COMFORMER SATISFY CRYSTAL PASSIVE SYMMETRIES
In this section, prove that the proposed invariant crystal graph representation is SE(3) invariant, and the proposed equivariant crystal graph representation is SO(3) equivariant, and both of them are periodic invariant. We use the same notations following the main paper as follows.
Crystal structures are represented by M = (A, P, L), where A = [a,,22 02 contains the d, -dimensional feature vectors for atoms the unit cell, contains the 3D Euclidean positions of atoms in the unit cell, and L [e, l2 La E R 3x3 describes the repeating patterns of the unit cell structure 3D space. Different from molecules and proteins, crystal structures naturally repeat infinitely in space. Formally, the infinite crystal structure can be described as
On E Rd xx
[pi. P2. p. E 3
P (p.lp: = p: + lo l1 + k222 + kals, k2, ha E EZ E Sign) n),
(8)
Á (á: -a Z, < n).
where the set P represents all 3D positions for every atom in the unit cell structure, and the set Å represents the corresponding atom feature vector for every atom.
A.2.1 SE(3) INVARIANT CRYSTAL GRAPH REPRESENTATION
As defined in the main paper, a crystal graph representation is SE(3) invariant if, for any rotation transformation R R 3x3 R| and translation transformation b R3 the crystal graph remains the same, f(A, P, = f (AA RP + b, RL). In the following, we prove the SE(3) invariance and periodic invariance step by step following the invariant crystal graph construction process.
16

---

Published as a conference paper at ICLR 2024
First, construct the lattice representation Ciis Cita Clis for every node, by using the process demonstrated in Sec.3.3. The selection of Cii, Cltz: Cliy is purely based on three components, including Euclidean distances, bond angles, and whether eii, Citz Giis form the right-hand system. And these three components are naturally SF(3) invariant which remain unchanged under any rotation transformation R E TR3x3 RR| = 1 and translation transformation b R Additionally, these three components will remain the same for a given node i and all its duplicates whatever unit cell structure you choose to represent the infinite crystal structure. Therefore, the choice of duplicates i àg, dg in the constructed lattice representation is SE(3) invariant and periodic invariant. Additionally, due to the fact that reflection transformations will change whether Cüi, Giiy Cii, form the high-hhand system, our proposed invariant crystal graph representation can distinguish chiral crystal structures.
Next, we use the minimum unit cell structure with n atoms and construct crystal graph with n nodes. This step is already handled by the JARVIS, MP, and MatBench datasets. Due to the fact that all the minimum unit cell structures for given crystal share the same number of atoms and corresponding atom features, this step is SE(3) invariant and periodic invariant.
Then, we build edges from neighboring nodes to every node i. Specifically, an edge will be built from node to node i when the Euclidean distance ||eyall2 between duplicate of j and i satisfies |leyill2 = - + k + +kg22 kgl2+Kige kigls pill2 I r, where E R is the cutoff radius. Duplicates i, ig. and i3 are included as neighbors of i to encode periodic patterns. Due to the fact that the Euclidean distance ||qrill2 between duplicate of and remains unchanged under SE(3) transformations and across different unit cell structure representations, and the choice of duplicates é i2, iz in the constructed lattice representation is SE(3) invariant and periodic invariant as shown in the first step, the neighborhood of node i is SE(3) invariant and periodic invariant.
In the end, we construct edge features for every edge. In the proposed invariant crystal graph, each edge has a corresponding feature vector {llerill2. Of Ui, 0f J. where 0z 0z'i, and Ofri,ts denoting the angles between еjι and {ell Gil,, eit, J. respectively. Due to the fact that ey and {eii, :liz: (ns } rotate accordingly to arbitrary rotation matrix R E 3 |R| = 1,, the relative bond angles 0 Ogri, and 0ft, remain unchanged under SE(3) transformations and across different unit cell structure representations. Thus, the construct edge features for every edge are SE(3) invariant and periodic invariant.
By combining these four steps in the invariant crystal graph construction process, we complete the proof that the proposed invariant crystal graph representation is SB(3) invariant and periodic invariant.
A.2.2 SO(3) EQUIVARIANT CRYSTAL GRAPH REPRESENTATION
As defined in the main paper, a crystal graph representation is SO(3) equivariant if, for any rotation transformation R E 3x33 R = 1 and translation transformation b R3 the crystal graph rotates S(A, RP + b, RL). In the following, we prove the SO(3) equivariance and periodic invariance step by step following the equivariant crystal graph construction process.
accordingly, R,,
For the lattice representation construction and the node and edge construction, it is the same the invariant crystal graph. Thus, the choice of duplicates i 22, 2 in the constructed lattice representation and the node and edge construction is SE(3) invariant and periodic invariant and can distinguish chiral crystal structures.
Then, for the edge features, |/enalla la is naturally SE(3) invariant and periodic invariant, while eys is SO(3) equivariant and periodic invariant.
Overall, the whole equivariant crystal graph representation is SO(3) equivariant and periodic invariant, with SE(3) invariant edge feature ||cjille la and SO(3) equivariant edge feature Cji-
Relationship between unit cell SE(() invariant and SO(3) equivariant functions. Here we show the relationship between unit cell SE(3) invariant and SO(3) equivariant functions. For any SO(3) equivariant function f, it satisfies R,S(A,P, L. S(A, RP + b, RL). Then, it can be seen that
17

---

Published as a conference paper at ICLR 2024
g ||fll2 is corresponding unit cell SE(3) invariant function. The proof is follows,
( R. + b, RL) =IIf(A, RP + +b, RL)I12
(9)
=|, P, L)
(10)
-I,
(11)
P,),,
(12)
Then the proof that g |lf /2 unit cell SE(3) invariant function is complete.
A.2.3 SE(3) INVARIANCE OF COMFORMER
iComFormer. The usage of SE(3) invariant crystal graph representations make the iComFormer SE(3) invariant. Thus, further proof is not needed.
eComFormer. The major difference between eComFormer and iComFormer is the usage of vector edge feature eji in node-wise equivariant updating layer. Due to the equivariant property of tensor product operations of TP1, 22 and invariant property tensor product operation TPu, after two tensor product layers in node-wise wise equivariant updating layer, the updated node feature f! is SE(3) invariant.
A.3 PROOFS OF BASE CASE AND INDUCTIVE STEP IN PROPOSITION
Definition 5 (Isometric crystal structures). Two crystal structures M (A, P, L) and M' (A' P' L') are isometric if 3 R E R R and E R3, such that
(Rp: blp: p; kil + kgl2 kgls, kn k2, Z Z. 1is. A' = À = {á. a; 2, 1 <i n}.
P'
where P are the infinite set representing all 3D positions for every atom in the two crystal structures and A, À' are the corresponding atom types as described in Sec. 2.1 and R + describes applying rotation transformation R and translation to every item the infinite set P.
A.3.1 PROOFS OF BASE CASE
Proof. We employ contradiction to prove it. We first assume there are two different infinite crystal structures with single atom in the unit cells (n = 1) that have the same SO(3) equivariant crystal graph or SE(3) invariant crystal graph, and then demonstrate this assumption leads to contradiction.
(1) Proof for SO(3) equivariant crystal graph. Following the SO(3) equivariant crystal graph construc- tion process in Sec. 3.3 we have obtained lattice representation (ei, Giig Gigg) where i represents the single atom in the unit cells, and é1. ig, is are three nearest duplicates of node satisfying right-hand system. Thus, eii, eily elt, form minimum lattice structure for these two crystals. Then, we can have
P (p:pp: pi hi kie kzen, kzenn ki, k2, ks E Z). P3 = (p 'p = pt kien, + kyen, + kyen, kn k2, ky E Z}.
(13)
(14)
where P. and P2 are the infinite for atom positions of these two different crystal structures. By using b = p Pi, we can have P P + b. Then, according Def. 5 it can be seen that these two previously assumed different crystal structures are identical, and it contradicts the assumption.
(2) Proof for SE(() invariant crystal graph. Following the SE(3) invariant crystal graph construction process in Sec. 3.2 we have obtained lattice representation {ea, Güg. Cna} where represents the single atom in the unit cells, and it, i2, i3 are three nearest duplicates of node i, satisfying right-handed system. Thus, Git, elty elig form minimum lattice structure for these two crys- tals. And duplicates 11, b2, and ig are encoded as neighbors of node i. Then, we have obtained { ile,, 8,,,,6, 8,,,, 8,,,, } as the edge feature from i {[|eiailla d, 0,zi,11 6,,,, as the edge feature from in to 1, and {[[e 2, dgg,, 0zgi 0igi, as the edge feature from ig to Since the SE(3) invariant crystal graph is identical for these two different crystals, these angles
18

---

Published as a conference paper at ICLR 2024
all identical to each other. Based on the Proof for SO(3) equivariant crystal graph above, we only need to prove the minimum lattice structures of these two crystals are identical, to contradict the assumption that these two crystals are different. Based on the edge features from node i, 82, to i, the minimum lattice structures of these two crystals can be recovered as follows.
For the first lattice vector Cii, and Ci for these two crystals, have
eii =||=,, x + 0y + Oz,
(15)
e/1 = 0y + Oz,
(16)
where x, z are unit vectors in a right-hand coordinate system, which means ||x||a = 1, llyl|e
1, ||z||2 I and y = 0, x 0, 2 = O, where denotes dot product. According to Def. 5 for this single atom case, all other choices of coordinate system can be converted to the above representations by applying rotation matrix R 33x3 R = 1.
For the second lattice vector eit, and e/t, for these two crystals, we have
et1_ = |e_.ll2cos )x + (e0.illasin(022itt + Oz,
(17)
111 = 0igil |2000 (0 in. )x |/iiil (otgil2ssim00:.. (0:g,, by + (z,
(18)
Again, according to Def. 5 all other choices to represent the second lattice vector can be converted to the above representations by applying a rotation matrix R E R3x3 |R| =.
Then, to prove that these two crystals share the same minimum lattice structure, we only need to prove the third lattice vector identical. For the third lattice vectors Cjig and e/,, for these two crystals, we have
Cjig
Gjis
Gita Ciis = /
(19)
Giig
Giig
Gis
Giiz |iiii [|2 |igii 2000
(0gg..
(20)
Giig
Cii
Gigg
Gii
|igi
1| lei 2000 20002000000000 (0igt,81);
(21)
Then we can have
eel
1l2 = | lll = ||e|,1l22
(22)
(ells e/is) eily 0,
(23)
(ells e(i,) ell, O,
(24)
which means (eiig e(i,) (identical), (eilg e ) = celi elig, where e E R and c + 0.
Additionally, due to the right hand system constraint during the graph construction process, we have
Cii x Ciiz
Ciaa > 0, eii elia eii 0.
(25)
(26)
Then we have
ell, x eli2 eils
(ellss e(in)) -ills = (||eity 113 - -e/i, -eilg) > 0,
(27)
eit x eit elts
(ells elia) e/i, eell e/i, | (12) >
(28)
(29)
then derive that
|
11 c/tg
Ciis
Ciis
citg
| l1
(30)
which contradicts that ||au, 11 = ||o// 1|. Thus, (Cigg c 0, and these two crystals have the identical minimum lattice structure. And this contradicts that these two crystals are different. Thus, the proof for SB(3) invariant crystal graph of Base Case is done.
19

---

Published as a conference paper at ICLR 2024
A.3.2 PROOFS OF INDUCTIVE STEP
Here we prove the relative position of newly added node is uniquely determined by the proposed SO(3) equivariant crystal graph and SE(3) invariant crystal graph.
Proof. We employ contradiction to prove it. We first assume there are two different relative positions of j.j that have the same SO(3) equivariant crystal graph or SE(3) invariant crystal graph, and then demonstrate this assumption leads to contradiction.
(I) Proof for SO(3) equivariant crystal graph. Following the SO(3) equivariant crystal graph con- struction process in Sec. 3.3 for any node in the m nodes already determined and i E NN,, we have edge feature etj- And the relative position of node and j' according to node i are P eij and Pjti = Cij, which contradicts the assumption. The proof is finished.
(2) Proof for SE(3) invariant crystal graph. Following the SE(3) invariant crystal graph construction process in Sec. 3.2 for any node in the m nodes already determined and i E NNs, we have edge feature {||ej| l2, dj oj ff }. Then, for the relative positions Pjj and pys of j and j'
according to i, we have
Pj ej |/e |2 ei |0os (0x.tt)) Pyi eih
(31)
Pji Ciiz = ||qa
22 eit c00 (( = Pyt Cliz
(32)
Pji Cia
|
2|
Cia
(c00s(0fi,1)) = Py' Ciig:
(33)
Then we can have
(py: - Pyt) Cüi, = 0,
(34)
(py: pyt) Giiz 0,
(35)
(Pyc - pyt) Glig = O,
(36)
and because ett, Git, Ut1, form set of basis for 3D coordinate system, we have (pj: Py( = 0. which contradicts the assumption. The proof is complete.
A.4 VERIFICATION OF GEOMETRIC COMPLETENESS
Geometrically complete crystal graphs can fully recover the input infinite crystal structures. We further verify the geometric completeness of our proposed SE(3) invariant crystal graph and SO(3) cquivariant crystal graph by reconstructing the crystal structures from graphs and then compare the reconstructed crystal structures with the original ones.
The reconstruction of infinite crystal structure using the proposed invariant crystal graph is decom- posed into four steps, including (1) the reconstruction of lattice representation for each node, (2) the reconstruction of neighboring node positions for given node, (3) the reconstruction of unit cell structure, and (4) the reconstruction of infinite crystal structure, as shown in Fig. g For step one, the information used includes node feature a; edge features ||ei, ll2. | ||2 ||etig 2. and i,g 01g5,61 iig and the right-hand system. For step two, the information used includes node feature aj, edge features |ejul 2. and 85,,8,, 0ft.tty 0ft,itg For step three, the information used is the fact that lattice representation is shared and aligned perfectly for a given crystal across all atoms. For step four, the information used is the constructed lattice representation in step one.
The reconstruction of infinite crystal structure using the proposed equivariant crystal graph can be done similarly. Specifically, in step one, the lattice representation for node can be directly reconstructed by using eti, - Giig. Gitg- And in step two, the position of the neighboring node j can be reconstructed by using eji- The step three and four can be done in the same way following the demonstration in Fig. g
We randomly use 500 crystal structures from the Materials Project, and convert them to the proposed SE(3) invariant and SO(3) equivariant crystal graphs using radius r determined by the 16-th nearest neighbors for every atom, and then use above reconstruction steps to recover the crystal structures from graphs. We then compare the structure differences between the recovered ones and the input ones using Pymatgen. As shown in Table. 7 the input crystal structures are fully recovered, with limited structure error (less than 10-6).
20

---

Published as a conference paper at ICLR 2024
Step One
Step Two
Step Three
Step Four
(1)
(2)
(3)
(4)
(5)
Figure 8: Illustrations of steps reconstructing infinite crystal structure using proposed crystal graph representation. The needed geometric information each step is marked by purple. (1) The edge information in crystal graph representation contains the edge length and three angles between three periodic lattice vectors. (2) Construct the three periodic lattice vectors, by only using the edge length place the first vector, and using the length and relative angle to place the second vector. For the third vector, two relative angles and the bond length are used. Here we choose the right-handed coordinate system for the periodic lattices, thus the orientation fully determined. (3) By using three angles and the bond length, the position of every neighboring node is determined. (4) Align the lattice vectors for neighboring node since they share the same repeating patterns and repeat step two. (5) Repeat step three until no node is left, and then recover the infinite structure using periodic lattice vectors.
We also extend the completeness verification to the dataset of T2, where crystal cell can have more than hundred atoms. We use the test set of dataset and convert crystals to the proposed SE(3) invariant and SO(3) equivariant crystal graphs using radius r determined by the 25-th nearest neighbors for every atom, and then use above reconstruction steps to recover the crystal structures from graphs. As shown in Table. g the input crystal structures, which can contain more than 700 atoms, are fully recovered with only 25 neighbors, with limited structure error.
Table Verification of geometric completeness of proposed SE(3) invariant and SO(3) equivariant crystal graphs by reconstructing crystal structures from graph representations.
Structure Differences
Rooted mean square deviation
Mean of largest point-wise distance
SE(3) invariant
3.28 10-7
4.87 10-7
SO(3) equivariant
3.03 * 10
4.48 * 10-7
Table Verification of geometric completeness of proposed SE(3) invariant and SO(3) equivariant crystal graphs by reconstructing crystal structures from graph representations T2 test set.
Structure Differences
Rooted mean square deviation
Mean of largest wise distance
SE(3) invariant
3.83 10 7
2.38 10
SO(3) equivariant
3.19 10
2.22 * 10 6
A.5
USAGE OF GROMETRIC INFORMATION IN COMFORMER
In this section, we discuss the usage of geometric information in iComFormer and eCom- Former. Specifically, we show that by the proper design of tensor product layers, the output Pruuddated of node-wise equivariant updating layer considers all two-hop bond angle information ΣjeN, EmEN, COS(Omgsi) in EjeN, TPo(f), (e()))).
21

---

Published as a conference paper at ICLR 2024
iComFormer. To begin with, the geometrically complete SE(3) invariant crystal graphs used as the imput of iComFrrmer. For the iCoomormer, the Euclidean distance |ejill is considered in the node-wise and edge-wise transformer layer, and three bond angles 0g 0j fji,is are considered in the edge-wise transformer layer. Therefore, iComFrrmer considers expressive geometric information of crystals during message passing.
eComFormer. The input of eComFormer is the proposed SO(3) equivariant crystal graph. The key question to answer is, how we consider expressive geometric information in eComFormer layers. The proof that the updated node feature S Suauutted considers bond angle information EjEN, EmEN, COS (ojjj)) is as follows.
First, EjEN, TPo(f). (eji)) a component in f'uupdoted: If E TPo(S), Yijej)) con- siders bond angle information EjEN, EmEN, COS(fmj.si) then the updated invariant node feature considers EjEN, EmEN, COS(Omjji).
We have
Σ TPo(ff.. Yi(e))) = Σ TP JEN
Σ
TPI( Y)(em))), (ej)))
(37)
JEN:
mEN,
TPo
Σ
TP YI(em))). Y1 (cj:))
(38)
WW
JEN;
mEN;
Σ
TPo
Σ
TP Y (eme))))
(39)
W
jeN:
mEN,
Σ
TP
Σ MEN,
Um Umi C1
(40)
WI
JEN.
where TP (emn)) Um and Pm conditioned on the invariant node feature f of node Ng, and learnable parameters (path weight) in TP1 Here for simplicity, only show the one output channel case of TP1, and in the implementation, the output can be multi-channel. Given two vector input V1 E R3, V2 E RR3 we have
[Po v2) =UTP. * (V1.x # V2.x V1, & V V1, 22
(41)
=VIPo COS (OSSO vl.22) ||ville ||v2|12,
(42)
where UTP, is a learnable scalar value in TPo, and COS describes cosine function. And we have [ev + V3, V2) = cTPolV1. v2) + TPo(V3, V2) due to the bi-linear property of TPp. Based these, we can further have,
Σ JEN,
TPo(f) Yi(eji) [N -
1
Σ
TPo
Σ mEN,
Um emjjj cjeji)
(43)
JEN.
Σ
VIP, Um C1 COS(@mij.s)
(44)
[
JEN, mEA;
Thus, ΣSEN, TPo Yeje))) considers all two hop bond angle information. And therefore it considers all angle information Pjiji. Ojjisi. fjaisi in the invariant crystal graph representation due to f2, Js E ENy. Beyond this, the node-wise transformer layer captures geometric information of |leji ll2, which further captures disturbs in pairwise Euclidean distances.
To sum up, the node-wise wise equivariant updating layer contains bond angle information between eji and ekj. where N and A Nj. and therefore considers all 2-hop angle information. And the node-wise transformer layer contains edge lengths |/eju|2 2 By combining them together, eComFormer considers expressive geometric information. And if lattice representation including i i2, ig is not encoded as the neighbor of node i, the infinite crystal structure, especially the periodic patterns, cannot be fully captured.
A.6 MODEL SETTINGS OF COMFORMER
We provide detailed model settings of iComFormer and eComFormer, and demonstrate the hyperpa- rameter settings of them for different datasets and tasks in this section.
22

---

Published as a conference paper at ICLR 2024
transformer
edge-wise
equivariant updating layer
fi
f
f
fi
fj
f
fil
fi
1:0g
Yo(áju)
(e(e)))
Ya(iji)
Unaaat
Concate
Ccacate
Concato
TP:
TP.
TP:
x
ex
Yo(4j))
(e(
Ya(ej:)
IP,
TP,
TP3
Trnneaard
BN Sigmord
Handeaadd
Honfwardd
BN Sigmoie
Mandward
LSN
m4g
maggi
(N)
(0.13)
Fepphtes
Figure 9: Illustration of the detailed network architectures of the proposed node-wise transformer layer, edge-wise transformer layer, and node-wise equivariant updating layer.
Detailed network architectures. We first show the detailed network architectures of the proposed node-wise transformer layer, edge-wise transformer layer, and node-wise equivariant updating layer in Fig. [ For all the experiments across three crystal datasets, we use two node-wise transformer layers, and one node-wise equivariant updating layer in between to form the eComFormer, and use one node-wise transformer layer, followed by one edge-wise transformer layer, and then - node-wise transformer layers to form the layer iComFormer.
Settings of embedding layers of ComFormer iComFormer and eComFormer both embed node feature a (atomic number) CGCNN embedding vector of length 92 and then mapped 256 dimensional initial node feature f by linear transformation. iComFormer and eComFormer both embed invariant edge feature ||ey: la to 0. 7..55/ eeelll2 and then increase vector by RBF kernels with 256 center value from -4.0 to 0.0, after that, both of them transfer the 256 dimensional vector initial edge feature fū. by a linear transformation layer and softplus activation layer. iComFormer embeds the invariant edge feature 0j Ogr,,igg and Ofte, to fy f3 and fi by taking the consine values and RBF kernels with 256 center values from -1.0 1.0 to 1.0, followed by a linear layer and a softplus activation layer. eComFormer embeds the equivariant edge feature ejii to spherical harmonics with rotation order 0, and 2 by using e3nn (Geiger & Smidt 2022b) library.
256 dimensional
Settings of node-wise transformer layer. LNo. LNK, LNy, LN are linear transformation layers that map 256 dimensional input features to 256 dimensional output fcatures. оk σv are the nonlinear transformations for key and value, including one linear layer that maps the concatenated 256 dimensional input features to 256 dimensional output features, one silu activation layer, and one linear layer that maps the 256 dimensional input features to 256 dimensional output features. maag the softplus activation layer.
Settings of edge-wise transformer layer. Similar to node-wise transformer layer, linear transfor- mations including LN L, L, LN L. LN, map 256 dimensional input features 256 dimensional output features. k of are the nonlinear transformations for key and value, including one linear layer that maps the concatenated 256 x dimensional input features to 256 dimensional output features, one silu activation layer, and one linear layer that maps the 256 dimensional input features to 256 dimensional output features. mig is the softplus activation layer.
Settings of node-wise equivariant updating layer. We directly use pherical harmonics Yo(ejı) Co. (ei) C1 * Cji E R and Yz(cja) E IR5 implemented in e3nn. LN transfers in- put node feature f: to f!" TPo(F Yo(eji) is the e3nn tensor product layer with rota- tion order 0 output features and 128 channels. (P) Y (eji)) and TP2()), Y (eji)) the e3nn tensor product layer with rotation order and 2 output features and 8 channels. TPo Yo(ej)), (f Y (ej))), (f,(2 Y2(ejj) are the e3nn tensor product layer with rotation order 0 output features and 128 channels. Tequi is the nonlinear transformation includes softplus layer, a linear transformation layer that maps 128 dimensional features to 256 dimensional features, and softplus layer. And LNequi is linear transformation layer that maps 256 dimensional input features to 256 dimensional output features.
23

---

Published as a conference paper at ICLR 2024
Table 9; Ablation studies, denotes the max rota- tion order of features in the equivariant layer.
Importance of higher rotation order features. As illustrated Table [ the inclusion of equivariant updating layer with features EjeN, (Po(f Y((e)) and ΣJEN, TPu(f Y2(ej)) with rotation orders one and two consistently enhances the performance of eComFormer from 30.2 to 28.4.
Method
node layer
equivariant layer
Test MAR
commerrmer eommformer eComFrreer
3
30.2
3
(o=
29.8
3
1 (o
28.4
Graph level prediction. After the transformer layers, aggregate the node features in the graph by mean pooling, and then use one linear layer to map the 256 dimensional graph level feature to 256 dimensional output feature, and then use silu activation layer, and then map the output scalar value by a linear transformation layer.
A.6.1 HYPERPARAMETER SETTINGS OF COMFORMER FOR DIFFERENT TASKS
In this subsection, we share the detailed hyperparameter settings of iComFormer and eComFormer for different tasks in these three crystal datasets. We slightly tuned the parameters of our methods, and higher performances are expected if hyperparameters are further tuned for different tasks.
Table 10: Model settings of iComerrmer for JARVIS.
node-wise transformer
edga-wise
Num. neighbors
formstion energy band gop (OPT) band po (MBJ) tocal meegg Ehull
0.001
25
0,0055
0.001
0.005
25
0.001
25
Table 11: Model settings eComFormer for JARVIS.
Num, mide-wise transformer
Numn node equisuriant updating
Leurning rale
Num. neighburs
formation energy
[mux onder=2)
0.011
25
band (OPT)
(max onder-2)
0,0055
25
(MHJ
anderw2)
0.001
energy
[max onden=2)
Elail
[max ander-2)
0.001
Table 12: Model settings of iComerrmer for the Materials Project.
Num. node-win wise transformer
Num, edge wise tranformer
Learning rale
Num. neighbers
formation energy
hand gop
0,0005
25
bulk modali
0.0001
16
0.0001
Table 13: Model settings of eComFormer for the Materials Project.
Num. node-wise transformer
Num. node wise equivariant updating
Learning
neighbors
formation
I (max onder-2)
0.001
25
gap sheur moduli
arderw2)
0.001
(max onder=?)
(max onder-2)
0.001
16
Table 14: Model settings of iComFormer for MatBench.
Num. mode wise transformer
Num. edge wise transformer
Learning rate
Num. neighbors
$4n7d
0.001
16
JARVIS. We show the model settings of iComFormer and eComFormer in Tables 10 and II
Specifically,, iComFormer is trained using MSE loss with Adam (Kingma & Ba 2015) optimizer
24

---

Published as a conference paper at ICLR 2024
Table 15: Model settings of eComFormer for MatBench.
Num, node-wise transformer
Nom. node wise equicariant updating
Learning cate
Num. neighibors
f.fom j0nZd
[max onder=2)
0.001
[max onfer-2)
0.001
16
for the tasks of MBJ bandgap and Ehull for 500 epochs, and formation energy for 700 epochs. with Onecycle (Smith & Topin 2018) scheduler and start of 0.3. For OPT bandgap and total energy, iComFormer are trained using LI loss with Adam (Kingma polynomial scheduler with starting learning rate 0.0005 and final learning rate 0.00001 for 800 epochs. eComFormer is trained using MSE loss with Adam (Kingma & Ba 2013) optimizer for all tasks for 500 epochs except OPT bandgap, with Onecycle (Smith & Topin 2018) scheduler and pet_start of 0.3. For OPT bandgap, eComFormer trained using L1 loss with Adam (Kingma Ba
Ba 2015) optimizer and
2015) optimizer and polynomial scheduler with starting learning rate 0.0005 and final learning rate 0.00001 for 800 epochs. The number of neighbors k indicates the k-th nearest distance we use as the radius for node i.
The Materials Project. We show the model settings of iComFormer and eComFormer in Table 12 and 13] Except bandgap of iComformer, all iComFormer and eComFormer models for the Materials Project are trained for 500 epochs using MSE loss with Adam (Kingma & Ba 2015) optimizer with Onecycle (Smith & Topin 2018) scheduler and pet_start of 0.3. For bandgap, iComformer is trained using [.1 loss with Adam (Kingma & Ba 2015) optimizer and polynomial scheduler with starting learning rate 0.0005 and final learning rate 0.00001 for 500 epochs.
MatBench. We show the model settings of iComFormer and eComFormer in Table and 15 All iComFormer and eComFormer models for the Materials Project are trained for 500 epochs using MAE loss with Adam (Kingma & Ba 2015) optimizer. And we use Onecycle (Smith & Topin 2018 scheduler and pet start of 0.3.
A.6.2 EFFICIENCY ANALYSIS WHEN INCREASING k
We provide further efficiency analysis for our proposed iComFormer and eComformer when increas- ing k. where & indicates that we use k-tth smallest distance from neighbors to the center to serve as the radius for node
We show the running time per epoch on the task of JARVIS formation energy for different ComFormer variants and different k values in Fig. 10 It can be seen that the running time increases linearly for all ComFormer variants when increasing k.
A.7 DATASET DETAILS
We provide more dataset details for the Materials Project, JARVIS, and MatBench in this section.
JARVIS. JARVIS-200 1001188-002..111 is first used by ALIGNN (Choudhary & DeCost 2021). D). We follow the experimental settings of Matformer (Yan et al. 2022) and PotNet (Lin et af.) 2023) and evaluate our methods on five important crystal property prediction tasks. The training, evaluating, and testing sets for formation energy, total energy, and bandgap(OPT) prediction tasks contain 44578, 5572, and 5572 crystals while contain 44296, 5537, and 5537 crystals for Ehull, and contain 14537, 1817, 1817 for bandgap(MBJ), We follow previous works and use mean absolute error as the evaluation metric. We directly use the benchmark results from Matformer (Yan et al. (002) and PotNet (Lin et al. (003). For the efficiency comparisons with previous works, we directly follow the efficiency comparison settings of Matformer (Yan et al., (202)) and use single RTX A6000 to test the running time per epoch. Among these crystal structures, 18865 crystals are experimentally observed.
The Materials Project. The Materials Project-2018.6.1 was first proposed and used by MEG- NET (Chen et al. 2219) collected from The Materials Project (Jain et al. 2013), but the methods are compared by using different random seeds and dataset sizes. To make a fair comparison, Yan et al. (2022) re- ve-train all baseline methods using the same data splits. We follow the experimental settings and data splits of Matformer (Yan et al. 2022) and PotNet (Lin et al. (003). Specifically, the formation energy and bandgap prediction tasks contain 60000, 5000, and 4239 crystals for training,
25

---

Published as a conference paper at ICLR 2024
eCorFoormer
160
iComForme(44) (ComFormer(3)
Rumning Time (s/epoch)
140
120
Radius determined by the k-th smallest distance
Figure 10: Efficiency analysis for iComFormer and eComformer when increasing k. where indicates the k-th smallest distance used as the radius for node i. We show the running time per epoch on the task of JARVIS formation energy.
Table 16: MP experimentally observed dataset.
Method
Number of training crystals
Test MAE
CGCNN (k=12)
29342
0.051
PDD-best (k=12)
29342
0.047
AMD
24067
0.78
iComformer (k=12)
24067
0.027
evaluating, and testing, respectively. And the shear and bulk moduli tasks contain 4664, 393, and 393 crystals for training, evaluating, and testing, respectively. Note that one validation sample in Shear Moduli is removed due negative GPa indicating the underlying unstable/mmtastabble crystal structure. The evaluation metric used is mean absolute eтгоr. We directly use the benchmark results from Matformer (Yan al. 2022) and PotNet (Lin al. 2023). Among these crystal structures, 30084 crystals are experimentally observed. We also further conduct experiments on the pure ex- perimentally observed crystal dataset with 30084 crystals, and use random splits 80%, 10%, 10% to form the training, evaluating, and test sets, and compare with CGCNN, PDD, and AMD as shown in
Table 16
MatBench. MatBench (Dunn et al., 2020) is a crystal property prediction benchmark, containing tasks with various data scales. To evaluate the robustness of our proposed methods, we conduct experiments on e_form with 132752 crystals and jdft2d with 636 crystals. It is worth noting that jdft2d a 2D crystal dataset, curated by Matbench. The traditional way (also in Matbench) to represent 2D structure is setting || llz > 10 Angstroms and not in the same plane of 41, 12. We directly the official MatBench package and guidelines to run these tasks, and report the benchmark results using metric mean absolute error (MAE) and root mean square error (RMSE) with standard deviation of five benchmark runs. The henchmark results of our methods will be updated to MatBench in the future. We directly the benchmark results from MatBench.
bars indicating the
A.8 DISCUSSIONS OF POTENTIAL CORNER CASES
In this section, discuss the potential corner about geometric completeness of the proposed crystal graphs, and potential corner cases due to the usage of the nearest neighbor algorithm, to aid future users when they deploy the proposed method for their applications.
26

---

Published as a conference paper at ICLR 2024
Group 2
For away
Far away.
K-5
Group 1
Figure 11: Demonstration of the hypothesis corner case that the constructed graph is not strongly connected. In this hypothesis case, the minimum unit cell structure contains two groups of atoms, marked by blue and pink. These two groups of atoms are far away from each other. We construct crystal graphs using radius determined by the k-th smallest Euclidean distances from neighbors to the center node. For this corner case, if the & = 5, there will be no connections between these two groups of nodes, resulting in not strongly connected graph. A solution to make the graph strongly connected is to increase the value and make & >> 5 (c.g., lt = will result a strongly connected graph).
A.8.1 POTENTIAL CORNER CASES OF GEOMETRIC COMPLETENESS
To achieve geometric completeness for crystalline materials, the only assumption we need is constructed crystal graph is strongly connected, which means for any node pair i,j, where j, represent the index of atoms in the unit cell, there is path from i to j.
the
Hypothesis corner cases. We then demonstrate the hypothesis cases that the constructed graph is not strongly connected and show how tackle this issue. As shown in Fig. 11 if there several groups that are far away from each other and the radius is not large enough, the constructed graph will not be strongly connected. is worth noting that for not strongly connected graph, no previous graph representations can achieve completeness, because of the fact that no geometric information between groups is captured.
Solution. Although this kind of hypothesis crystalline material may not exist in reality, a solution for this Hypothesis case will be to increase the radius or increase the value of & (when you determine radius r by the k-th smallest Euclidean distances from neighbors to the center node).
A.8.2 POTENTIAL CORNER CASES OF USING NEAREST NEIGHIBRRS
When a reference node is needed, a widely used E(3) invariant way to choose the reference node using Euclidean distance and nearest neighbors. Similar to SphereNet (Liu al., (022) and ComENet (Wang et al. 2022) which employ the nearest neighbor algorithm to choose reference nodes, our method also uses the Euclidean distance based nearest neighbor algorithm determine the lattice representations to achieve rotation and translation invariance.
Potential corner case. When several periodic duplicates of the atom in a given crystal share the same Euclidean distances, e.g., ||en, ll2 |ea, ll = |eu, ||2, these duplicates will share the equal possibility to selected as the reference node.
Solution and discussion. This issue occurs for algorithms using nearest neighbors, including SphereNet (Liu et al. 2022), ComENet (Wang et al. 2022), CGCNN (Xie & Grossman 2018), and GATGNN (Louis et al. 2020). If the nearest neighbor algorithm employed is deterministic, there will be a unique graph for the corner case. the nearest neighbor algorithm employed is not deterministic, these several different representations of the corner will occur with equal possibility during the training process (similar to data augmentation).
If the user plans to further restrict the result of the nearest neighbor algorithm, additional information can be used to select from these candidates with the same Euclidean distances. In any case, this issue will not influence the completeness of the constructed representation.
27


Reviews:
[{"review_id": "gItayfKZkg", "reviewer": "Reviewer_pQH7", "summary": "This paper proposes a graph transformer network with rotation, translation, and periodic equivariance/invariance for learning on crystal graphs. It introduces a crystal graph representation with geometric completeness and equivariance. On top of it, the paper also proposes a transformer architecture with efficiency and expressivity. Experiments on three different benchmarks show the superiority of the proposed network.", "questions": "- I don't have specific questions.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper introduces a crystal graph representation with equivariance and geometric completeness together with an equivariant transformer architecture, which tackles a practical problem in material analysis.\n- The authors have provided adequate theoretical proof.\n- The authors have presented extensive experiments on different benchmarks, efficiency analysis, and ablation studies, showing the effectiveness of the proposed network with superiority over prior works.\n- The illustrations and explanations in the paper are clear and intuitive. I don't have any background in material science, but the problem statement in the paper sounds reasonable to me.", "weaknesses": "- I have limited knowledge about crystal graph learning, but the arguments in the paper look convincing to me. The authors mentioned learning higher-order properties in their own discussion of limitations. This could possibly be solved by the SO(3)-representation with sperical harmonic basis [1, 2].\n\n[1] Thomas, Nathaniel, et al. \"Tensor field networks: Rotation-and translation-equivariant neural networks for 3d point clouds.\" arXiv preprint arXiv:1802.08219 (2018).\n\n[2] Fuchs, Fabian, et al. \"Se (3)-transformers: 3d roto-translation equivariant attention networks.\" Advances in neural information processing systems 33 (2020): 1970-1981.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kYFgZxIsp1", "reviewer": "Reviewer_v1jy", "summary": "The paper described an experimental approach to material property prediction by using graphs built on periodic crystal structures.", "questions": "The words \"crystal graph\" appear a few times before section 2.2 with references but without details. In Definition 1, could the authors please explain the meaning of \"a crystal graph\" and the symbols \"=\" and \"->\"? \n\nDid the paper consider the invariance of crystal descriptors under permutations of atoms?  \n\nDoes a manually chosen cutoff radius (line 3 from the bottom of page 4) make any invariant incomplete because perturbing atoms can scale a unit cell larger than any fixed radius?\n\nSince the paper title promised complete transformers, have the authors found any geometric duplicates in the used datasets? Some duplicates were reported by Widdowson et al (MATCH 2022) as being investigated by several journals for data integrity. \n\nThe dataset details in section A.5 could include the most important information: whether crystals are simulated or experimental, and if the \"ground-truth\" properties are obtained from computer modeling or physical experiments. \n\nIt seems that all used crystals are hypothetical with simulated properties. Is the main contribution a computer program that predicts the outputs of other computer programs? \n\nIf initial property computations were too slow and most likely iterative, why not to stop their computations earlier instead of writing another program simulating the simulations?\n\nHow many hidden parameters and CPU hours were used for producing the experimental results?\n\nWhat results in the paper are stronger than the past work below?\n\n(1) continuity under perturbations in Theorem 4.2 and generic completeness of the density fingerprint in Theorem 5.1 from Edelsbrunner et al (SoCG 2021),  \n(2) continuity under perturbations in Theorem 4.3 and generic completeness with an explcit reconstruction in Theorem 4.4 from Widdowson et al (NeurIPS 2022),\n(3) full completeness in Theorem 10 from Anosova et al (DGMM 2021), extended in arxiv:2205.15298,  \n(4) 200+ billion pairwise comparisons of all periodic crystals in the Cambridge Structural Database completed within two days on a desktop, see the conclusions in Widdowson et al. \n\nDid the authors know about the classical results in crystallography cited above, starting from Niggli (1927), Lawton (1965), and Parthe (1987)?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "The authors should be highly praised for studying important real objects such as solid crystalline materials and for attempting to give formal Definitions 1-4. The paper is generally well-written and contains enough details that helped to understand the difficulties.", "weaknesses": "The word \"problem\" appears only once (in the abstract), though a rigorous and explicit problem statement might have helped to understand the unresolved challenges. \n\nIf we need a complete and invariant description of a periodic crystal, crystallographers solved this problem nearly 100 years ago by using Niggli's reduced cell of a lattice and then recording all atoms in so-called standard settings, see the book \"TYPIX standardized data and crystal chemical characterization of inorganic structure types\" by Parthé et al, which actually applies to all periodic crystals, not only inorganic. \n\nHowever, all these standardizations have become obsolete in the new world of big and noisy data because the underlying lattice (not even a unit cell) of any periodic crystal is discontinuous under almost any perturbation, which is obvious already in dimension 1.\n\nFor example, the set Z of all integers is nearly identical to a periodic sequence with points 0, 1+ep_1, ..., m+ep_m in the unit cell [0,m+1] for any sma;; ep_1,...,ep_m close to 0, though their minimal periods (or unit cells) 1 and m+1 are arbitrarily different. \n\nThis discontinuity was reported for experimental crystals already in 1965, see Lawton SL, Jacobson RA. The reduced cell and its crystallographic applications. Ames Lab., Iowa State Univ. of Science and Tech.\n\nA more recent example from Materials Project shows two nearly identical crystals whose unit cells differ by a factor of (approximately) 2\nhttps://next-gen.materialsproject.org/materials/mp-568619\nhttps://next-gen.materialsproject.org/materials/mp-568656\n\nMoreover, atoms in any material always vibrate above absolute zero temperature, so their positions continuously change. As a result, any crystal structure with fixed atomic coordinates in a database is only a single snapshot of a potentially dynamic object, especially for proteins whose structures are also determined often by crystallization. \n\nHence the new essential requirement for any (better than the past) representations of crystals is a proved continuity under perturbations of atoms. This problem has been solved by Widdowson et al (NeurIPS 2022) with theoretical guarantees and practical demonstrations on the world's largest collection of materials: the Cambridge Structural Database. The underlying invariants have a near-linear time in the motif size and were used for property predictions by Ropers et all (DAMDID 2022) and by Balasingham et al (arxiv:2212.11246). \n\nDespite the authors citing one of the papers above, Definition 1 didn't follow the definitions from the past that modeled a crystal as a periodic point set not as a graph. \n\nAny graph representation of a crystal or a molecule is discontinuous because all chemical bonds are only abstract representations of inter-atomic interactions and depend on numerous thresholds on distances and angles, while atomic nuclei are real physical objects.\n\nThe traditional representation in (1) and Definition 5 (appendix A.3) of an isometry order atoms in a unit cell. While many crystals contain identical atoms, the invariance under permutations of atoms is missed because the paper has no words \"permute\" or \"permutation\".\n\nThe permutation invariance seems lost when using angles, which depend on point ordering. \n\nThe property prediction makes sense only for really validated data, not for random values, because neural networks with millions of parameters can predict (overfit) any data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yGNoOI99PK", "reviewer": "Reviewer_ZKHH", "summary": "Geometric graph representation learning for crystal material property prediction is challenging due to the periodic and infinite nature of crystals. This paper proposed SE(3) invariant and SO(3) equivariant crystal graph representations for geometric completeness of crystalline materials. Furthermore, two variants of a SE(3) transformer is proposed for crystalline material property prediction, i.e., iComFormer that uses invariant geometric descriptors of Euclidean distances and angles, and eComFormer that uses equivariant vector representations.\n\nExperiments are conducted on three widely-used crystal benchmarks: JARVIS, the Materials Project, and MatBench. Experimental results showed that ComFormer achieves the best prediction accuracy across a range of crystal properties in most cases. The ablation study verifies the importance of geometric completeness of crystal graphs and higher rotation order features.", "questions": "Please kindly refer to the weakness part. I'd like to discuss with the authors for the above weaknesses part.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. $\\textbf{Problem Formulation}$: This paper identified an existing issue in previous related works, i.e., they map different crystal structures with different properties to the same graph representation and thus produce wrong property predictions. This is a good motivation. Based on this motivation, this paper aims to resolve this issue by proposing a new graph representation for crystalline materials to distinguish different crystals and remain the same under crystal passive symmetries.\n\n2. $\\textbf{Method}$: The proposed SE(3) invariant graph construction is sound and the subsequent crystal transformer networks are effective in crystal property prediction.\n\n3. $\\textbf{Experimental Results}$: The performance is superior than or comparable to the compared approaches across all three datasets in most cases.", "weaknesses": "1. $\\textbf{Method}$\n\n1.1 Is it just the way of lattice representation construction (Section 3.2 and Section 3.3) that introduces SE(3) invariant and SO(3) equivariant properties, or the proposed Transformer network (Section 4) has such properties? In other words, if the proposed Transformer network is applied to an ordinary graph (say a k-NN graph), does the SE(3) invariant and SO(3) equivariant properties still hold?\n\n1.2 It seems that the node-wise transformer layer in SE(3) invariant message passing (Section 4.1) has no significant difference from the conventional graph Transformer (or graph attention network) [1]. Though detailed design may be different, but the general structure/scheme are similar. Correct me if I'm wrong.\n\n1.3 It is unclear why edge feature $\\textbf{e}_{ji}$ needs to be embedded using spherical harmonics, in second paragraph of Section 4.2. A justification is needed.\n\n1.4 \"$\\textbf{TP}_0,\\textbf{TP}_1,\\textbf{TP}_2$ are tensor product layers\" in the fourth paragraph of Section 4.2, What is tensor product layer and why need it here? Could you provide a concise and intuitive explanation?\n\n2. $\\textbf{Experiments}$\n\n2.1 Is it possible to provide some qualitative experimental results (visualization) of SE(3) invariant property and/or SO(3) equivariant property, to prove the graph representation and the proposed Transformer network indeed learns such properties? It is not mandatory, but just out of curiosity.\n\n2.2 In Table 5, the size of eComFormer is larger than iComFormer by two to three times. Any explanation or reason for this model size?\n\n3. $\\textbf{Writting}$\n\n3.1 The paper spends a lot of space for background introduction and graph representation description, leading to just 1.5 pages for Transformer network design. It lacks the description of the rationale behind the network design, especially for SO(3) message passing, which causes a bit confusion in finding enough novelty of the proposed network.\n\n$\\newline$\n\n[1] Veličković P, Cucurull G, Casanova A, Romero A, Liò P, Bengio Y. Graph Attention Networks. ICLR, 2018.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hERmw5snT3", "reviewer": "Reviewer_HGjX", "summary": "Constructing graphs that effectively capture the complete geometric information of crystals remains an unsolved and challenging problem. In order to address it, this paper introduced a novel approach that utilizes the periodic patterns of unit cells to establish the lattice-based representation for each atom, enabling efficient and expressive graph representations of crystals.", "questions": "1) How to deal with the crystal whose lattice contains (0,0,0)? Do the datasets in experiments contain this type of data? In theory, the method proposed by the author cannot handle this situation. Why was it not explained in experiments?\n\n2) What is the role of SE(3) invariant and SO(3) equivariant? In what scenarios should eComFormer be used, and in what scenarios should iComFormer be used?  What are the reasons for iComFormer outperforming eComFormer in most cases?\n\nSuggestions:\nIf possible, it is recommended that the author add some experiments that demonstrate the completeness of the representation.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1) The presentation of this paper is good. The author's statement is logical, and the vocabulary used is relatively easy to understand.\n\n2) This paper proposes an interesting problem. A complete crystal representation is important for downstream tasks.", "weaknesses": "1) This method is not universal as it cannot handle crystals with lattice containing (0,0,0). The paper mentions that “Third, we select the duplicate with the next smallest distance ||e_ii3||_2, and verify that eii3, eii1 and eii2 are not in the same plane in 3D space, and repeat until eii3 is found”, but if lattice containing (0,0,0), then eii3 cannot be found.\n\n2) The experiment is not comprehensive enough. The most significant contribution of this paper is that it proposes a complete representation, but the final experiment only verifies that it can better predict without verifying the completeness of the representation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)