Periodic Graph Transformers for Crystal Material Property Prediction
Authors: Keqiang Yan
Yi Liu
Yuchao Lin
Shuiwang Ji
Venue: NeurIPS 2022 Accept (2022)
Keywords: Periodic Invariance
periodic pattern encoding
material transformers
crystal property prediction
material discovery

Abstract: We consider representation learning on periodic graphs encoding crystal materials. Different from regular graphs, periodic graphs consist of a minimum unit cell repeating itself on a regular lattice in 3D space. How to effectively encode these periodic structures poses unique challenges not present in regular graph representation learning. In addition to being E(3) invariant, periodic graph representations need to be periodic invariant. That is, the learned representations should be invariant to shifts of cell boundaries as they are artificially imposed. Furthermore, the periodic repeating patterns need to be captured explicitly as lattices of different sizes and orientations may correspond to different materials. In this work, we propose a transformer architecture, known as Matformer, for periodic graph representation learning. Our Matformer is designed to be invariant to periodicity and can capture repeating patterns explicitly. In particular, Matformer encodes periodic patterns by efficient use of geometric distances between the same atoms in neighboring cells. Experimental results on multiple common benchmark datasets show that our Matformer outperforms baseline methods consistently. In addition, our results demonstrate the importance of periodic invariance and explicit repeating pattern encoding for crystal representation learning. Our code is publicly available at https://github.com/YKQ98/Matformer.

Periodic Graph Transformers for Crystal Material Property Prediction
Keqiang Yan Computer Science & Engineering Texas A&M University College Station, TX 77843
Yi Liu'
Computer Science Florida State University Tallahassee, FL 32306 liuy@cs. edu
keqiangyan
Yuchao Lin
Shuiwang Ji
Computer Science & Engineering Texas A&M University College Station, TX 77843
Computer Science & Engineering Texas A&M University College Station, TX 77843 sji@tamu. edu
kruskallin@tami edu
Abstract
We consider representation learning on periodic graphs encoding crystal materials. Different from regular graphs, periodic graphs consist of minimum unit cell repeating itself on regular lattice in 3D space. How to effectively encode these pe- riodic structures poses unique challenges not present regular graph representation learning. In addition to being E(3) invariant, periodic graph representations need be periodic invariant. That is, the learned representations should be invariant shifts of cell boundaries as they are artificially imposed. Furthermore, the periodic repeating patterns need be captured explicitly as lattices of different sizes and orientations may correspond to different materials. In this work, we propose a transformer architecture, known as Matformer, for periodic graph representation learning. Our Matformer is designed to be invariant to periodicity and can capture repeating patterns explicitly. In particular, Matformer encodes periodic patterns by efficient use of geometric distances between the same atoms in neighboring cells. Experimental results on multiple common benchmark datasets show that results
Matformer outperforms baseline methods consistently. In addition, demonstrate the importance of periodic invariance and explicit repeating pattern encoding for crystal representation learning. Our code is publicly available https: btttp://github. com. /KKg////t///oMeerrrrrrr
1
Introduction
Crystal material property prediction is important for the discovery of new materials with desirable
properties 38 37
50 52
33 6. Different from molecules and proteins [5]
51
42
9
26 which are commonly represented as graphs 45 29
25
27
34 crystals
consist of minimum unit cell repeating itself on a regular lattice in 3D space. Thus, crystals are naturally represented as periodic graphs. A key challenge of crystal material property prediction
lies in how to effectively encode periodic structures that are not present in regular molecular graph 13
representations [10 20 41 23 22
47 E(3) invariance for molecular graphs requires the representation for given molecule be invariant to translation, rotation and reflection
31
24 39
Equul senior contributions
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

E2
0 E1
Figure 1: Illustrations of periodic patterns and the multi-edge graph construction method. Green lines are artificial boundaries to form one possible unit cell that repeats in infinite space for the given crystal. (a). An illustration of periodic patterns in 2D space. We use blue and red arrows show how the blue atom repeats itself along & and Eg. We show a general case in nature that E and &z are not orthogonal. In this specific example, 0 is less than 5. (b) and (c). Illustrations of the multi-edge graph construction method and the constructed graph. The green circle shows atom i's neighborhood N₁. and black arrows are edges from atom i's neighbors itself. As atom j repeats twice within i's neighborhood in this example, the multi-edge graph construction method builds two edges from atom j to atom í. The erystals are 3D structures in practice, and we use illustrations in 2D for simplicity.
transformations 3D space. Beyond that, periodic graphs of crystals require unique periodic invariance. Periodic invariance has two facets; those are, the learned representations should be invariant to both the scaling up of the minimum repeatable unit cells and shifts of periodic boundaries. Although the former has been considered in Xie et al. [52 [33 the later rarely identified explicitly and sometimes not considered by previous studies (52 46
Furthermore, the repeating patterns of periodic graphs should be captured explicitly. Given a fixed minimum unit cell structure, lattices of different sizes and orientations may correspond to different materials. Without such periodic patterns, the infinite structures of crystals may not be represented accurately. However, the explicit encoding of periodic patterns is not explored in previous studies [52]
53
33 54
33 D..
In this work, we propose tackle periodic graph representation learning by incorporating both periodic invariance and periodic pattern encoding. We propose a periodic graph transformer, known as Matformer, that is periodic invariant and can capture periodic repeating patterns explicitly for crystal representation learning. Matformer achieves periodic invariance through two uniquely designed graph construction methods. It further encodes periodic patterns by efficient use of geometric distances between the same atoms in neighboring cells, thereby capturing the lattice size and orientation of a given crystal. We conduct experiments on two commonly used material benchmark datasets, including the Materials Project [16] and Jarvis (8). Results show that Matformer outperforms baseline methods consistently on various tasks. In addition, our results demonstrate the importance of both periodic invariance and periodic pattern encoding for crystal representation learning.
Background
Crystal property prediction and crystal structures. Given a crystal represented as (A, P. L). crystal property prediction aims to predict a target property value y. which is either real as y E R or categorical as y E 21,2 .J. for regression or classification task with C classes, respectively. Specifically, a crystal is represented as unit cell with periodic patterns. A unit cell is a minimum repeatable structure for the given crystal, and it can be described by matrices A and P. A E de is the dimeennnlal
(a1, a2 a,,'T E Raxxa is the atom feature matrix, where feature vector for atom the unit cell. P (p. P2. -P, 1" E Rax3 is the position matrix, where P; E 3 contains the Cartesian coordinates for atom i in 3D space. To further encode periodic patterns, an additional lattice matrix I. = L2, f3)² E R3333 used to describe how a unit cell repeats itself in three directions, including L1, l2, and lz. We show a 2D case of periodic patterns for easy illustration in Fig. (a). Note that crystals usually possess irregular shapes in practice. Hence. li, l2 and E3 are not always orthogonal in 3D space. Formally, given crystal representation

---

(.,,L), the infinite crystal structure can be represented as
P (p. = P; + k 2 + 2222 + kzL3, k1 22, n), À (á; a,, E Z, S i n).
Here, P contains all possible positions for each atom associated with the same a; in Å.
Multi-edge graph construction for erystals. The multi-edge graph construction proposed by Xie and Grossman [52] aims to capture atom interactions across cell boundaries, which are imposed artificially. In regular molecular graph, a node corresponds to single atom. In contrast, in a multi-edge graph, node i represents atom and all its duplicates in the infinite 3D space. Apparently, node i contains the atom features vector a, and all positions the set {p. pp; - P; P.+ki& 1 k2222 kzL2, k1, k2, k3 2). Formally, the multi-edge graph construction method builds edges between nodes as follows. Given prefixed radius IR, if there exists any 3-tuple (K Kig Kg), where ky, ky, K2 E R, such that the Euclidean distance dz. IR satisfies d,s = ||p, + kj + k222 + kzLs Pilla r. an edge is built from j to i with the initial edge feature d/z- An example of the edge construction is shown in Fig. (b). Intuitively, if there exist m positions of node within the radius of the center node i, this method builds edges from node j to node i. By considering all possible positions of every node within predefined radius in 3D space, the multi-edge graph construction method can essence capture atom interactions across cell boundaries 40
3 Periodic invariance and periodic pattern encoding for crystals
Different from molecular graphs, crystal graphs consist of a minimum unit cell repeating itself on regular lattice in 3D space. When encoding such periodic structures, unique challenges lie in periodic invariance and periodic pattern encoding. In this section, we propose formally define and analyze the importance of these two components.
3.1 Periodic invariance for crystals
Periodic invariance is proposed based on E(3) invariance, which is defined below.
Definition 1 (Unit Cell E(3) Invariance). A function f: P,L, + X' is unit cell E(3) invariant such that for all Q gax3 [Q] =1 and E IR we have f(A, = f(A, QP + b, QL), where Q is rotation and reflection transformations, and b is translation transformations in 3D space.
Intuitively, the structure of a cell remains the same when either applying rotations and reflections to position matrix P and lattice matrix L together, or applying translations to only. Correspondingly, the output of the unit cell E(3) invariant function should remain the same.
In addition to unit cell E(3) in- variance, periodic invariance is also shown necessary for generat- ing valid crystal representations. Specifically, when the lattice ma- trix L [2, f2, Ls) E 33x3 is fixed for a crystal, we can still obtain different position matri- ces E nx and different unit cell structures by shifting the pe- riod boundaries. As shown in Fig. (a) and (b), the formed unit cell structures are different for the same crystal by shifting pe- riod boundaries. To this end, we further introduce periodic invari- ance, which shows that when the periodic boundaries are shifted or scaled up, the periodic invariant representation should remain the same. Formally, based on Sec.
Pi Pa
(b)
Illustration of periodic invariance. Purple lines are the edges between nodes inside a unit cell. Red points the corner points of the unit cells. For example, cells in (a) and (b), respectively. (a) and (b) show different unit cells describing the same crystal, caused by shifting the period boundaries along x axis from P1 to P2- By comparing (a) and (b). we show a graph construction method that breaks periodic invariance.
Figure 2:
and po are for unit
further define a function & PA + PA simulating how to form different unit cells
3

---

i4, + Ealla
1162
Co
(a)
(b)
(c)
Figure 4: Illustrations of the used radius-based graph construction method and the proposed peri- odic pattern encoding for Matformer. Black arrows are the formed edges by radius-based graph construction and brown arrows are self-conmecting edges. (a). Illustration that the used radius-based graph construction satisfies periodic invariance. (b). Illustration of periodic pattern encoding using self-conmecting edges. On the left, we show the designed self-connecting edges to encode the lattice matrix L [21 l2, Lz) S Rax3 in 3D space. We design six self-connecting edges ||L1//2, || ||6/||2. [[ + [g] 2. |[ + fa 12. ll + fal12. The geometric shape of L can be determined by these show the added self-connecting edges in 2D space for easy illustration. (c). Illustration of the constructed graph with periodic pattern encoding for the above 2D case.
six edges. On the right,
from a given infinite crystal structure. For an infinite crystal structure represented as (A, P). & uses corner point and shape matrix L to form unit cell represented as (A, P). In addition, we use a E N to indicate the scaling up of a repeating unit cell formed by periodic boundaries. Then the formal definition of periodic invariance is below.
Definition 2 (Periodic Invariance). unit cell E(3) invariant function f (A, + X is periodic invariant if (A, P. f( aL, aL) holds for all p E R2 and E N4.
Significance of periodic invariance. Breaking periodic invariance will result in different crystal graphs for the same crystal. In Fig. (a) and (b), we show a graph construction method that breaks periodic invariance. This method is employed by transformer-based model, known as Graphormer [54), which first uses radius to include all the atoms of interest in nearby cells and then builds fully connected graph. A detailed illustration is shown in Appendix. A.1. Based on the formal definition of periodic invariance, in this study, we aim to integrate such an important component in our Matformer. By doing this, our model is able to construct a distinct crystal graph for a given crystal structure, resulting in a more informative and discriminative crystal learning scheme.
3.2 Periodic pattern encoding
As introduced in Sec. L= Lsl e R3x3 containing periodic patterns is another key component describe crystal structures. Essen- tially, periodic patterns show how the minimum repeatable structure (A, P) expands itself in in- finite 3D space. Without such periodic pattern encoding in L, crystal structures are treated finite structures similar to molecules. As shown Fig. 3 the widely used multi-edge graph con- struction method [52] 4 [1 44 6) only captures local interactions among atoms but ignores the important periodic patterns. However, such pe- riodic repeating patterns need to be captured explicitly, as lattices of different sizes and ori- entations may correspond to different materials, Hence, To better represent the infinite structures of crystals, we argue that the periodic patterns
(b) Figure 3: Illustration that periodic patterns are not encoded in multi-edge graph. Grey lines show the captured topology information by the multi-edge graph construction. We use three radius circles because there are three atoms in unit cell. (a). Il- lustration of the multi-edge graph construction. (b). Illustration that the multi-edge graph method only captures local geometric information but ignores periodic patterns for the infinite structure. L 3x3x3 should be explicitly taken into consideration in crystal learning.

---

4 The proposed Matformer
4.1 The proposed graph construction methods
In this section, we introduce the proposed graph construction methods for Matformer. Our methods effectively integrate periodic invariance and periodic pattern encoding.
Invariant crystal graph construction. We consider two crystal graph construction methods, includ- ing radius-based graph construction and fully connected graph construction. Both of them satisfy periodic invariance, and mathematical proof's can be found in Appendix. A.2. In Fig. 2 we show by example that treating atoms as single nodes breaks periodic invariance. Instead, our methods follow the fashion introduced in Sec. 2 to treat node as atom i and all its repeated duplicates.
We use the multi-edge graph construction [52 introduced in Sec. 2 as alternative crystal graph construction method. Note that although the multi edge graph construction satisfies periodic invari- ance in Fig. 4 (a), several existing works 222 B3 do not follow the settings exactly thus breaking periodic invariance. Within the given radius shown in Fig. (a), these studies form the neighborhood of node by selecting í nearest neighbors ranked by geometric distances. If there exist several different atoms with the same distance to node i, there is no deterministic way to select from them, as shown in Appendix. 1. As a result, periodic invariance cannot be guaranteed.
Given the fully-connected fashion employed in Graphormer has achieved impressive performance on molecular learning, we further propose another graph construction method for Matformer, known as the fully connected graph construction. This method uses a different strategy to determine neighbors for each center node. Specifically, for node i and j, it builds edges for the entries corresponding to the t smallest distances in {d,y (d, Ipp. P, k; L1 k2l2 kg ll2,, ki, K2 Kg E Z). It can be seen every pair and j is connected in the constructed graph and there are t edges between them.
Overall, the used multi-edge graph construction and proposed fully-connected graph construction both satisfy the important periodic invariance. Particularly, they possess great flexibility to be used in future studies for crystal learning. In this study, we employ both methods as part of our proposed Matformer, and in main experiments, the radius-basse method is used due to better empirical performance.
Periodic pattern encoding with self-connecting edges. In this study, we propose to encode the important [e, Ez. E [3333 into crystal graphs by adding self-connecting edges. As mentioned in Sec. 3.2 periodic patterns describe sizes and orientations of lattices of a crystal structure, eventually determining the properties of this crystal. A natural step for encoding such repeating periodic patterns is to consider the relative positions between an atom and its nearby repeated duplicates. Formally, given an atom i with position p, and L [&i, l2, lz)= E R3x3 we need to encode the atom's 's three nearby duplicates with positions P, + L1, P. + l2. and p, + Lz. It widely known that a direction vector f, is determined by both its length |/ and orientation. Essentially. ||e/|2 indicates the geometric distance between atom and its corresponding duplicate. However, the computing of orientation information, such as angles, usually induces high complexity. Hence, it is not practical to encode such orientation information into transformer architectures. To this end, we propose to use geometric distances solely to implicitly consider the orientation information.
Specifically, we use additional distances to determine angles between any two direction vectors in L. For example, the angle between l and &g can be easily computed by ||1] ||22l22. and an additional distance [|e + E2/12. Hence, based on these three distances, we can determine lengths of e and &z, and the relative orientation between them. Extensively, we use six geometric distances, including ||e: ||22||2. ||&/]22. [|e + 22112, ||e + lall2, and ||e| + Lall2 in our study, shown in Fig. 3 (b). By doing this, the length of cach direction vector and the angle between any two direction vectors can all be determined. As result, the shape formed by lattice matrix L is then fixed. Overall, we build the affrrementioned six geometric distances as six self-connecting edges for node i. By doing this, our model is capable of encoding periodic patterns in L completely, resulting in more accurate crystal representation learning scheme. Importantly, our approach also guarantees periodic invariance.
Overall, the graph construction for Matformer consists of two necessary stages, including invariant graph construction and periodic pattern encoding. For the first stage, we rigorously prove that the multi-edge graph construction satisfies periodic invariance in Appendix. A.2 and show that several previous works [52 [x] break periodic invariance using a different neighbor selection strategy.
Additionally, we propose a fully-connected crystal graph construction method satisfying periodic
5

---

Output
Linear
Readout
Update
Update
Aggregate
SiLU
mn;
m.
Linear
Average Pooling
Aggregate
Concatenate Lineur
Lincar
((BN) m72
Layernor
Head cad2
Matformer
Linear
Layer
Layerorr
Hadamard
Σ JEN:
Sigmoid
Matforemr
Lincar
Layer
Hadamard
Lincar
Softolus
Q
K
Lincar
Linear
RBF
CGCNN
LNQ
LNK K
LNv
LN
kernels
embedding
fi fj
fr fj
(a)
(b)
Figure 5: Illustration of detailed Architecture of Matformer. The overlapping graphics are used to denote different attention heads. (a). Illustration of Matformer pipeline. (b). Illustration of the detailed Matformer layer in (a). We show the case with two attention heads for simplicity.
invariance, and the method could be used in future studies for crystal representation learning. For the second stage, we naturally encode periodic patterns into constructed graphs by adding self-connecting edges without breaking periodic invariance. A constructed graph in 2D case is shown in Fig. (c).
4.2 Message passing scheme
Building constructed graphs introduced in Sec. 4.1
propose message passing scheme for
Matformer. Formally, we denote a constructed crystal graph as = (A, E). Here, each a, E A the d.-dimensional feature vector for atom i, as introduced Sec. 2 Particularly, e E is de-dimensional feature vector for the h--th edge between nodes i and We Tollow the regular attention mechanism that computes query, key, and value 330 28). Our proposed message passing scheme is composed of three steps; those are, edge-wise attention coefficients computing, edge-wise value message computing, and node updating. Formally, we let f denote the input feature vector of node i for the L-th layer Matformer. The message passing scheme of the ε-th layer described as below.
In the first step, d'y. kis and at for the h-th edge between and j are computed as
( hs = LLNK(FT) k (N,( e = =
d ok"
(2)
d = (q.1 KY = (kilkyle,) a
da
where LNQ, LNk, and Lyg denote the linear transformations to compute query, key, and edge embedding 2-th layer, respectively. eÿ is the intermediate output for ety We use and to denote Hadamard Product and concatenation. Note that g²; is the concatenation three q, vectors to match the dimension of k By doing this, when computing aty q: attends each of k;, kj. and et for integrating more information in attention.
Particularly, omit the softmax to enhance the model's 's capability distinguish nodes with different degrees, and to make the whole network more efficient.
After obtaining coefficients axi in the second step, compute m that is the message of e as
0; (f fj (fj")) m sigmoid(LNorm((r.)) LN upppdae (0./oy (l)lllye)))) {
(3)

---

where LNv and upddate are the linear transformations to compute value and the updated message, and LNorm denotes the layer normalization operation.
Finally, in the third step, we compute node i's feature vector f. Specifically, we first obtain message m; by aggregating information from node i's neighborhood over multiple edges, then achieve fi as
=
ΣE JEN: h
LNorm (m J LN + ( (BN(m:))
(4)
where a is the used activation function, and BN indicates batch normalization. In addition, LNmsg and LNiea are linear transformations to update the messages on edges and the old atom features.
Graphormer represents an effective transformer variant for molecular graph learning. The differences between Graphormer and the proposed Matformer lie in both graph construction and message passing scheme. Firstly, Graphormer treats every atom as a single node and breaks periodic invariance, as mentioned in Sec. 3.T] For the message passing. Graphormer uses the node-wise attention and encodes pairwise distances as attention bias. It cannot work properly multi-edge graphs for crystals. While Matformer is specifically designed for multi-edge crystal graphs by performing edge-wise attention and encoding geometric information into edge-wise messages, described above. The detailed architecture of our Matformer is shown in Fig. 5
Related work
Crystal property prediction. Several existing methods 46
18
19.
14
model crystals as chemical
consider 3D structures and formulate crystals as 3D graphs, then apply GNNs to learn from crystal graphs. As crystals are essentially periodically repeated structures, the graph construction needs consider periodic invariance and periodic pattern encoding. There are limited efforts to identify these two unique components. As an early work, CGCNN [52] proposes to capture atom interactions across artificial cell boundaries by using multi-edge graphs described in Sec. - The multi-edge graph satisfies periodic invariance as described in Sec. 3.1 but fails consider the important periodic patterns, as described in Sec. 3.2 The multi-edge graph construction method is widely used in the following studies 40 33
formulas and employ sequence models to process them. Other studies 522
40
33
4
41
6
M. Based on the constructed crystal graphs. many GNN variants have been proposed for effective crystal representation learning 52
40
33
41 6 I.. Specifically,
Nequip [ considers E(3) equivariance for materials, and satisfies periodic invariance using multi- edge graphs, but fails to capture periodic repeating patterns. We also notice recent work [48] for periodic graph generation, which considers periodic graphs as finite graphs and breaks periodic invariance. Recently, ALIGNN [6] achieves the best performance on two major material datasets. uses angle information the message passing to generate more informative and discriminative representations. However, the use of angles introduces excessive time complexity.
Geometric GNNs and graph transformer. Many efforts have been made to incorporate 3D geomet- ric information in molecular learning. Exemplary studies include SchNet [1], DimeNet [3 22]
SphereNet 31 GemNet (24] etc. However, these methods are designed for molecules without peri- odic patterns. Recently, graph transformers [54] using geometric information, e.g., Graphormer [54].
have shown great potential on real-world graph data. However, Graphormer considers neither periodic invariance nor periodic pattern encoding.
Differences with method. To the best of our knowledge, periodic invariance and periodic pattern encoding described in Sec. 3 are rarely identified and explored in existing works for crystal property prediction. CGCNN [52] breaks periodic invariance on some corner cases because uses twelve nearest neighbors determined only by distances as described in Sec. 4.1 In addition, previous methods including CGCNN [52]. SchNet 4T.. MEGNET 4. CYATT 40, GATGNN [33, NEQUIP [I] and ALIGNN 6 all fail to consider the important periodic pattern encoding as introduced in Sec. 3.2 Especially, following GAT [45) GATGNN 33 employs very limited kind of attention mechanism
that is not conditioned on query, as explained in GATv2 2 [2]. As result, the model capacity is reduced compared with the self attention mechanism employed in Matformer. In addition, the usage of softmax limits the capability of GATGNN of distinguishing nodes with different degrees, as
mentioned in Sec. 4.2 For Graphormer [54], although it achieved remarkable success on the Open
Catalyst Challenge BI, the employed graph construction method breaks periodic invariance when
7

---

Table I: Comparison in terms of test MAE on The Materials Project dataset. To make the comparison clear and fair, We show results from retrained models using exactly the same training, validation, and test sets. Results from original papers are shown in Appendix A.5. The best results are shown in bold and the second best results shown with underlines.
Formation Energy
Band Gap
Bulk Moduli
Shear Moduli
Method
eVatoo
eV
log(GPa)
log(GPa)
CGCNN
[52
0.031
0.292
0.047
0.077
SchNet
41
0,033
0.345
0,066
0.099
MEGNET
0.030
0.307
0.060
0.099
GATGNN
33
0.033
0.280
0.045
0.075
ALIGNN
0.022
0.218
0.051
0.078
Matformer
0.021
0.211
0.043
0.073
applied to crystals. Compared with Graphormer, Matformer is specifically designed for crystals considering both periodic invariance and periodic patters..
Experimental studies
6.1 Experimental setup
We conduct experiments on two material benchmark datasets, including The Materials Project [66 and JARVIS 8 The detailed descriptions for The Materials Project and JARVIS datasets are shown in Appendix. A.3. Baseline methods include CFID [7). CGCNN [52], SchNet [4], MEGNET 4. GATGNN 33], and ALIGNN (). Unless otherwise specified, for all the baseline methods, we report the results taken from the referred papers or provided by original authors. All Matformer models are trained using the Adam optimizer (2I] with weight decay [32] and one cycle learning rate scheduler 43 We only slightly adjust learning rates from 0.001 and training epochs from 500 for different tasks. Detailed Matformer configurations for different tasks are provided in Appendix. A.4.
6.2 Experimental results
The Materials Project. We first use The Materials Project-2018.6.1 dataset which contains 69239 crystals, evaluate Matformer. We notice that previous works [52 each other either using datasets of different sizes, or using datasets with the same size but splitting the datasets with different random seeds. To make the comparison clear and fair, we retrain all corresponding models using exactly the same training, validation and test sets across all methods and report the results in Table. T To avoid confusion, we still put original results from referred papers in Table. [ insside parentheses. For retrained bascline models, we provide the detailed configurations in Appendix. A.5. The used metric is test MAE following previous studies [52]
6 compare with
33 6).
It can be seen from Table. T that Matformer achieves the best performances on all tasks consistently by significant margins. Specifically, reduces the formation energy by 4.5% of the second best model, which is a significant margin. Furthermore, for Bulk Moduli and Shear Moduli tasks with only 4664 training samples, Matformer achieves the best performances, indicating Matformer's adaptive ability to tasks of small training scales.
JARVIS dataset. The quantitative results for Jarvis shown in Table. 2 Matformer outperforms the baseline methods significantly on all of these five tasks. Compared with ALIGNN, Matformer has stronger discriminative ability due explicit encoding of periodic patterns. Specifically, Matformer reduces Jarvis Ehull by 0.012, which is 15.8% of ALIGNN. Furthermore, Matformer achieves the best performances for Bulk Moduli and Shear Moduli the Mateirals Project with 4664 training samples, and Bandgap(MBBJ) in JARVIS with 14537 training samples, indicating its adaptive ability tasks of various data scales. Overall, the superior performances show the effectiveness of periodic pattern encoding in our Matformer message passing. In addition, compared with ALIGNN, our Matformer is more efficient. We evaluate the efficiency of Matformer by comparing with ALIGNN using JARVIS formation energy dataset. The mean time of ten runs for training and inference using best model configurations of ALIGNN and Matformer are reported. We also report the total number of parameters of each model. In Table. B we show that Matformer is three times faster than ALIGNN
8

---

Table 2: Comparison between Matformer and other baselines in terms of test MAE on JARVIS dataset. The best results are shown in bold and the second best results are shown with underlines.
Formation Energy
Band
Total Energy
Ehull
Bandgap(MBI))
Method
eVattom
eV
eV/atom
eV
eV
CFID 7
0.14
0.30
0.24
0.22
0.53
CGCNN
52
0.063
0.20
0.078
0.17
0,41
SchNet
41
0.045
0.19
0,047
0.14
0,43
MEGNET
0.047
0,145
0.058
0.084
0.34
GATGNN 33
0.047
0.17
0,056
0.12
0,51
ALIGNN 6
0.0331
0,142
0.037
0.076
0.31
Matformer
0.0325
0.137
0.035
0.064
0.30
Table 3: Efficiency comparison with ALIGNN on Jarvis Formation Energy task. We show the training time per epoch, total training time, inference time for the whole test set, and total number of parameters.
Models
Time/epoch
Total
Inference
Model Para.
ALIGNN
327 s
27.3 h 8.9 h
156
15.4 MB
Matformer
64 s
59
11.0 MB
in total training time and near three times faster in inference time, for the whole test set. Matformer is also much lighter than ALIGNN in terms of model size.
Energy within Threshold. Following OC20 3. we use energy within threshold (EwT), which measures the percentage of estimated energies that are likely to be practically useful when the absolute error is within certain threshold, to evaluate Matformer's s capability for periodic graph learning. This metric new, but is well recognized by the community as it useful practice. Due to significant performance gaps between ALIGNN and other baseline methods on formation energy and total energy for these two datasets in terms mean absolute error (MAE), we only compare Matformer with ALIGNN. Table. shows that Matformer outperforms ALIGNN consistently for all three energy prediction tasks. Interestingly, the performance gains of our Matformer beyond ALIGNN in terms of EWT mainly come from with JARVIS, the Materials Project has 15422 more traning samples. As a result, the percentage of predicted energies obtained by Matformer within 0.01 increases by 14.69%, which is much better than ALIGNN, revealing the huge potential of Matformer when larger crystal dataset is available.
accurate energy predictions within absolute error of 0.01. Compared
6.3 Ablation studies
In this section, we demonstrate the importance of periodic invariance and explicit repeating pattern encoding for crystal representation learning by ablation studies. We also evaluate the building blocks particularly designed for our Matformer. Specifically, we conduct experiments on JARVIS formation energy, and the test MAE is used as the quantitative evaluation metric. We also provide ablation studies on the use of sigmoid and layernorm instead of softmax in Matformer layer in Appendix A.6.
Periodic invariant graph construction. We demonstrate the importance of periodic invariant graph construction by comparing radius multi-edge graph, denoted as Radius, with the graph construction method proposed by Graphormer, denoted as OCgraph, on the exactly same Matformer architecture. Note that the constructed crystal graphs by OCgraph are super large, containing more than n.² edges,
Table 4: Comparison between Matformer and ALIGNN in terms of EWT on JARVIS Formation Energy, JARVIS Total Energy and The Materials Project Formation Energy. We use EwT (0.02) to mark the threshold of 0.02 and EwT (0.01) to mark the threshold of 0.01. The best results are hold.
Formation MP
Formation JARVIS
Total JARVIS
Method
EwT (0.01)
EWT (0.02)
EWT (0.01)
EWT (0.02)
EWT (0.01)
EWT (0.02)
ALIGNN
49.94%
71.10%
39.59%
59.64%
35.09%
55.20%
Matformer
55.86%
75.02%
41.17%
60.25%
36.84%
57.36%

---

Table Ablation studies on periodic invariance and periodic pattern encoding. We use OCgraph to denote graph construction method proposed by Graphormer. PI denotes periodic invariance and PE denotes periodic encoding.
Graph
PI
PE
layer
head
batch
Test MAE
OCgraph
32
0.0530
Radius w/o PE
V
X
3
1
32
0.0348
Radius w/o
X
64
0.0337
T-fully w PE
5
4
64
0.0402
Radius w PE
4
64
0.0325
where n is the atom number in a cell. We adjust the Matformer configurations to train these large graphs on single RTX A6000 GPU. can be seen from Table. [ that when using OCgraph to our Matformer, the test MAE drops dramatically of 53% because of breaking periodic invariance, compared with radius-based multi-edge graphs Matformer. We also compare two periodic invariant graph construction methods described in Sec. 4.1 We denote fully connected graph with smallest pairwise distances as T-fully, and use
The result shows that Radius is better than T-fully.
Encoding of repeating patterns. We denote periodic pattern encoding as PE. In Table. 5 we show that omitting the periodic pattern encoding results in significant drop of test MAE from 0.0325 to 0.0337, revealing the importance of periodic patterns for crystal representation learning.
Complexity of introducing angular informa- tion. Dropping angular information largely im- proves running efficiency of Matformer com- pared with ALIGNN. We show that for the orig- inal crystal graph with nodes and 6n edges, the corresponding line graph with angles will have 6n nodes and 66n edges, leading high computational cost. The detailed complexity analysis of adding angular information are provided in Appendix. A.7. Additionally, provide the running time and performance analysis of Matformer with angle information in Table, [6] We use two Matformer layers process extra angular information, and use Radial Basis Function kernels (6) and Spherical Bessel Functions with Spherical Harmonics [23 31 24 to encode angles, denoted Matformer + Angle RBF and Matformer + Angle SBF. Table. shows that introducing angular information will increase both the training time per epoch and in total by around 3 times, without much performance gain. This may due the periodic invariant graph construction and periodic patterns encoding in Matformer already capture sufficient information to identify crystal structures.
Table 6: Ablation studies angular information. We show the training time per epoch, total training time, and test MAE.
Models
MAF
Time/epoch
Total
Matfonner
.0255
Matforner Angle SBF
.0332
173
Matforner Angle RBF
.0325
165
22.9
7 Conclusions and discussions
In this work, first propose to formally define periodic invariance and periodic pattern encoding for periodic graph learning. We then propose Matformer for periodic graph representation learning, which is invariant to periodicity and can capture repeating patterns explicitly. Experimental results on common benchmark datasets show that our Matformer outperforms baseline methods consistently. In addition, results demonstrate the importance of periodic invariance and explicit periodic pattern encoding for crystal representation learning. One potential direction beyond this work is to include angular information properly satisfy both periodic invariance and to encode periodic patterns with relatively low time complexity, and this is one limitation of our work. Besides, negative societal impacts of material discovery may apply to our work.
Acknowledgments and Disclosure of Funding
We thank Tian Xie for answering our questions on CGCNN, This work was supported in part by National Science Foundation grant IIS-2006861.
10

---

References
[1] Simon Batzner, Albert Musaclian, Lixin Sun, Mario Geiger, Jonathan P Mailoa, Mordechai Kornbluth, Nicola Molinari, Tess E Smidt, and Boris Kozinsky. E (3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials. Nature communications, 13(1):
1-11, 2022.
[2] Shaked Brody, Uri Alon, and Eran Yahav. How Attentive are Graph Attention Networks? In International Conference on Learning Representations, 2021.
[3] Lowik Chanussot*, Abhishek Das*, Siddharth Goyal*, Thibaut Lavril*, Muhammed Shuaibi*, Morgane Riviere, Kevin Tran, Javier DeranoD Caleb Ho, Weihua Hu, Aini Palizhati, Anuroop Sriram, Brandon Wood, Junwoong Yoon, Devi Parikh, C. Lawrence Zitnick, and Zachary Ulissi. Open Catalyst 2020 (OC20) Dataset and Community Challenges. ACS Catalysis, 2021. doi: 10. 10.1021/acscatal.004555
[4] Chi Chen, Weike Ye, Yunxing Zuo, Chen Zheng, and Shyue Ping Ong. Graph Networks as Universal Machine Learning Framework for Molecules and Crystals. Chemistry of Materials, 3(9))3354-33572 2019.
[5] Zhantao Chen, Nina Andrejevic, Tess Smidt, Zhiwei Ding, Qian Xu, Yen-Ting Chi, Quynh T Nguyen, Ahmet Alatas, Jing Kong, and Mingda Li. Direct Prediction of Phonon Density of States With Euclidean Neural Networks. Adv. Sci, 8:2004214, 2021.
[6] Kamal Choudhary and Brian DeCost. Atomistic Line Graph Neural Network for improved materials property predictions. npj Computational Materials, 7(1):1-8, 2021.
[7] Kamal Choudhary, Brian DeCost, and Francesca Tavazza. Machine learning with force-field- inspired descriptors for materials: Fast screening and mapping energy landscape. Physical review materials, 2(8):083801, 2018.
[8] Kamal Choudhary, Kevin F Garrity, Andrew CE Reid, Brian DeCost, Adam J Biacchi, Angela R Hight Walker, Zachary Trautt, Jason Saitrick-Simpers, Gilad Kusne, Andrea Centrone, et al. The joint automated repository for various integrated simulations (JARVIS) for data-driven materials design. npj Computational Materials, 6(1):1-13, 2020.
[9] Alex Fout, Jonathon Byrd, Basir Shariat, and Asa Ben-Hur. Protein Interface Prediction using Graph Convolutional Networks. Advances in Neural Information Processing Systems, 30, 2017.
[10] Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling. SE (3)-transformers: 3d roto-translation equivariant attention networks, Advances in Neural Information Processing Systems, 33: 13:199011981, 2020.
[11] Hongyang Gao and Shuiwang Ji. Graph U-Nets. In International Conference on Machine Learning, pages 2083-2092. PMIR, 2019.
[12] Hongyang Gao, Yi Liu, and Shuiwang Ji. Topology-Awee Awae Graph Pooling Networks. JEEE Transactions on Pattern Analysis and Machine Intelligence, 43(12):4512-4518 2021.
[13] Mario Geiger and Tess Smidt. SKKi::2007.09453. 2022.
e3nn:
Euclidean Neural Networks.
arXiv preprint
[14] Rhys EA Goodall and Alpha A Lee. Predicting materials properties without crystal structure: Deep representation learning from stoichiometry. Nature communications, 11(1):1-9, 2020..
[15] Marco Gori, Gabriele Monfardini, and Franco Scarselli. A new model for learning in graph do- mains. In Proceedings, 2005 IEEE international joint conference on neural networks, volume 2, pages 729-734, 2005.
[16] Anubhav Jain, Shyue Ping Ong, Geoffroy Hautier, Wei Chen, William Davidson Richards, Stephen Dacek, Shreyas Cholia, Dan Gunter, David Skinner, Gerbrand Ceder, et al. Commentary: The Materials Project: materials genome approach to accelerating materials innovation. API. materials, 1(1):011002, 2013.

---

[17] Priyank Jaini, Lars Holdijk, and Max Welling. Learning Equivariant Energy Based Models with Equivariant Stein Variational Gradient Descent. Advances in Neural Information Processing Systems, 34, 2021.
[18] Dipendra Jha, Logan Ward, Arindam Paul, Wei-keng Liao, Alok Choudhary, Chris Wolverton, and Ankit Agrawal. Elemnet: Deep learning the chemistry of materials from only elemental composition. Scientific reports, 8(1): 1-1):1-13, 2018.
[19] Dipendra Jha, Logan Ward, Zijiang Yang, Christopher Wolverton, Ian Foster, Wei-keng Liao, Alok Choudhary, and Ankit Agrawal. IRNet: A general purpose deep residual regression framework for materials discovery. In Proceedings of the 25th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pages 2385-2399) 2019.
[20] Nicolas Keriven and Gabriel Peyré. Universal invariant and equivariant graph neural networks. Advances in Neural Information Processing Systems, 32, 2019.
[21] Diederik P Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In Proceedings of the 3rd International Conference on Learning Representations, 2015.
[22] Johannes Klicpera, Shankari Giri, Johannes T Margraf, and Stephan Günneemann. Fast and uncertainty-aware directional message passing for non-equilibrium molecules. arXiv preprint 1XXiv:201.114115 2020.
[23] Johannes Klicpera, Janek GroB, and Stephan Günnemann. Directional Message Passing for Molecular Graphs. In International Conference on Learning Representations, 2020.
[24] Johannes Klicpera, Florian Becker, and Stephan Günnemann. GemNet: Universal Directional
Graph Neural Networks for Molecules. In Advances in Neural Information Processing Systems, 2021.
[25] Meng Liu, Hongyang Gao, and Shuiwang Ji. Towards Deeper Graph Neural Networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, ACM, 2020.
[26] Meng Liu, Youzhi Luo, Limei Wang, Yaochen Xie, Hao Yuan, Shurui Gui, Haiyang Yu, Zhao Xu, Jingtun Zhang, Yi Liu, et al. DIG: a turnkey library for diving into graph deep learning research. Journal of Machine Learning Research, 22(240):1-9, 2021.
[27] Meng Liu, Keqiang Yan, Bora Oztekin, and Shuiwang Ji. GraphEBM: Molecular graph generation with energy-based models. arXiv preprint arXiv:2102.00546. 2021.
[28] Yi Liu and Shuiwang Ji. CleftNet: Augmented Deep Learning for Synaptic Cleft Detection from Brain Electron Microscopy. IEEE Transactions on Medical Imaging, 40112):357733518 2021.
[29] Yi Liu, Hao Yuan, Lei Cai, and Shuiwang Ji. Deep Learning of High-order Interactions for Protein Interface Prediction. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 679-687,20 2020.
[30] Yi Liu, Hao Yuan, Zhengyang Wang, and Shuiwang Ji. Global Pixel Transformers for Virtual Staining of Microscopy Images. IEEE Transactions Medical Imaging, 396))2256-226. 2020.
[31] Yi Liu, Limei Wang, Meng Liu, Yuchao Lin, Xuan Zhang, Bora Oztekin, and Shuiwang Ji. Spherical message passing for 3d molecular graphs In International Conference on Learning Representations, 2021.
[32] Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. In International Conference on Learning Representations, 2018
[33] Steph- Yves Louis, Yong Zhao, Alireza Nasiri, Xiran Wang, Yuqi Song, Fei Liu, and Jianjun Hu. Graph convolutional neural networks with global attention for improved materials property prediction. Physical Chemistry Chemical Physics, 22(32):1114111811 2020.
12

---

[34] Youzhi Luo, Keqiang Yan, and Shuiwang Ji. GraphDF: A Discrete Flow Model for Molecular Graph Generation. In International Conference on Machine Learning, pages 7192-7203, PMLR, 2021.
[35] Bryce Meredig, Ankit Agrawal, Scott Kirklin, James E Saal, Jeff Doak, Alan Thompson. Kunpeng Zhang, Alok Choudhary, and Christopher Wolverton. Combinatorial screening for new materials unconstrained composition space with machine learning. Physical Review B, 89(9):094104, 2014.
[36] Anton Oliynyk, Erin Antono, Taylor D Sparks, Leila Ghadheigi, Michael W Gaultois, Bryce Meredig, and Arthur Mar. High-throughput machine-learning lerrning-dnee synthesis of full-heusler compounds. Chemistry of Materials, 28(20):7324 7331, 2016.
[37] Paul Raccuglia, Katherine C Elbert, Philip DF Adler, Casey Falk, Malia B Wenny, Aurelio Mollo, Matthias Zeller, Sorelle Friedler, Joshua Schrier, and Alexander Norquist. Machine- learning-assisted materials discovery using failed experiments. Nature, 533(7601):73-776 2016.
[38] Rampi Ramprasad, Rohit Batra, Ghanshyam Pilania, Arun Mannodi Kaanodd-Kanakkithodii and Chiho Kim. Machine learning in materials informatics: recent applications and prospects. npi Computational Materials, 311) 2017.
[39] Victor Garcia Satorras, Emiel Hoogeboom, Fabian Bernd Fuchs, Ingmar Posner, and Max Welling. E (n) Equivariant Normalizing Flows. In Advances in Neural Information Processing Systems, 2021.
[40] Jonathan Schmidt, Love Pettersson, Claudio Verdozzi, Silvana Botti, and Miguel A. L. Marques. Crystal Graph Attention Networks for the Prediction of Stable Materials. Science Advances, 7 (49):aa 2021. doi: 10. 126/sciadv..bb7949..
[41] Kristof Schütt, Pieter-Jan Kindermans, Huziel Enoc Sauceda Felix, Stefan Chmiela, Alexandre Tkatchenko, and Klaus-Rober Müller. SchNet: A Continuous-filee Convolutional Neural Network for Modeling Quantum Interactions. Advances in Neural Information Processing Systems, 30, 2017.
[42] Nino Shervashidze, Pascal Schweitzer, Erik Jan Van Leeuwen, Kurt Mehlhorn, and Karsten M Borgwardt. Weisfeiler-lemman graph kernels. Journal Machine Learning Research, 12(9), 2011.
[43] Leslie N Smith and Nicholay Topin. Super-convergence: Very fast training of residual networks using large learning rates. 2018.
[44] Jonathan M Stokes, Kevin Yang. Kyle Swanson, Wengong Jin, Andres Cubillos-Ruiz Nina M Donghia, Craig R MacNair, Shawn French, Lindsey A Carfrae, Zohar Bloom-Ackermann et al. Deep Learning Approach to Antibiotic Discovery. Cell, 180(4):688-700 2020.
[45] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph Attention Networks. In International Conference on Learning Representations, 2018.
[46] Anthony Yu-Tung Wang, Steven K Kauwe, Ryan J Murdock, and Taylor Sparks. Composition- ally restricted attention-basee network for materials property predictions. Npj Computational Materials, 1-1):1-10, 2021.
[47] Limei Wang, Yi Liu, Yuchao Lin, Haoran Liu, and Shuiwang Ji. Comenet: Towards complete and efficient message passing for 3d molccular graphs. arXiv preprint arXiv:2206.08515, 2022.
[48] Shiyu Wang, Xiaojie Guo, and Liang Zhao. Deep Generative Model for Periodic Graphs. arXiv preprint arXiv:2201. 1932, 2022.
[49] Zhengyang Wang, Meng Liu, Youzhi Luo, Zhao Xu, Yaochen Xie, Limei Wang, Lei Cai, Qi Qi, Zhuoning Yuan, Tianbao Yang, et al. Advanced Graph and Sequence Neural Networks for Molecular Property Prediction and Drug Discovery. Bioinformatics. 38(9):2579-255 2586, 2022.
13

---

[50] Logan Ward, Ankit Agrawal, Alok Choudhary, and Christopher Wolverton. A general-purpor machine learning framework for predicting properties of inorganic materials, npi Computational Materials, 2(1):1-7, 2016.
[51] Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S Pappu, Karl Leswing, and Vijay Pande. MoleculeNet: a benchmark for molecular machine learning. Chemical science, 9(2):513-530 2018.
[52] Tian Xie and Jeffrey C Grossman. Crystal Graph Convolutional Neural Networks for an Accurate and Interpretable Prediction of Material Properties. Physical review letters, 120(14): 145301, 2018.
[53] Tian Xie, Xiang Fu, Octavian Octavian-Eugen Ganea, Regina Barzilay, and Tommi Jaakkola. Crystal Dif- fusion Variational Autoencoder for Periodic Material Generation. In International Conference on Learning Representations, 2022.
[54] Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and The-Yan Liu. Do Transformers Really Perform Badly for Graph Representation? Advances in Neural Information Processing Systems, 34, 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Conclusion and limitation section.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix. A.2.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed reproduce the main ex- perimental results (either in the supplemental material or as a URL)? [Yes] We have included the detailed settings of our experiments. We will publish our implementation upon acceptance.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix. A.4 and Appendix. A.5.
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [N/A] Error bars are not necessary for baseline methods and our method.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] All of these are reported in Appendix. A.4.
4. If you are using existing assets (e.g., code, data, models) or curating/relcasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
14

---

5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "tJ4luCl2umg", "reviewer": "Reviewer_ifKM", "summary": "In this paper, the author/s proposed a graph transformer architecture, named Matformer, to infer the physical properties of crystal materials in which their atomic representation is encoded as graphs. The Matformer architecture encodes repeating patterns and is invariant to periodicity; these two properties are fundamental in modeling crystal materials. The performance are assessed on standard benchmarks, considerably outperforming the baselines. ", "questions": "My questions are reported in the weaknesses", "limitations": "I wondered about the model's performance when no periodicity or repetitive patterns in the graph are present.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:** \n\n*Originality*\nProposing a model that is invariant to pattern periodicity and that it explicitly captures repetitive patterns in a graph is a novelty.\n\n\n*Quality*\nThe quality of this work is good; almost all proofs are present in the paper or in the appendix. The experimental part is clear and the reference adequate.\n\n*Significance*\nI think this work is significant for the community since it addresses specific properties of graphs that are not commonly studied.\n\n**Weaknesses:** \n* One of your direct competitors is Graphormer, but there are no comparisons, why?\n* In line 183 you said \"... rigorously prove ...\" , where is the proof?\n* In line 46 you used $\\mathbb{N}$ for the categorical case... the natural number is infinite, do you have infinite classes?\n* In Eq 2 the definition of $q_{ij}^h$ consists of the concatenation of $q_i$ three times... is it correct?\n* In line 215, you said that one of the differences with respect to Graphormer is that \" ...  Graphormer treats every atom as a single node.\", but you are doing the same following the definition of $a_i \\in \\mathbf{A}$. Can you elaborate more?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "qWTUs78AOO0", "reviewer": "Reviewer_oomT", "summary": "Periodic graph is ubiquitous in the real world, such as crystal material. Representation learning on periodic graphs is a significant task for downstream tasks such as property prediction. This paper proposes a new transformer-based framework for the representation learning on periodic graphs. Specifically, it comes up with a new strategy to construct graphs and a way to encode lattice matrix L. The experiments section shows the effectiveness of the method.", "questions": "I would suggest the author address the questions mentioned under the \"Weakness\" in the \"Strengths And Weaknesses\" section of the review. ", "limitations": "The limitations of the model are not mentioned, the negative societal impacts of your work are marked as N/A.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n1. The paper formularizes the periodic graph in a clear way using A, P and L matrix.\n3. The experimental results show the effectiveness of the proposed method.\n\nWeakness:\n1. The motivation of the paper can be further improved:\n    a) Why do we need to construct a graph from the unit-cell coordinates rather than directly using coordinates to conduct representation learning? What's the benefit to borrow the graph structure here? I believe there are a few works for representation learning on 3D positions such as [1], [2] and [3]\n    b) What is the motivation to form self-connecting edges to contain the information of lattice matrix? Why not just concatenate it with the representation of a single unit cell?\n2. It seems that section 3.1 and 3.2 can be moved to section 2 or they can just serve as a single section, since it seems that they are the motivation/theoretical ground of the proposed model not the model itself.\n3. Line 45, the paragraph is titled as \"Crystal property prediction\" but it seems that the whole paragraph is about the definition of peridic lattice. The title might be modified accordingly.\n4. It seems that E(3) invariance has been mentioned even in the abstract but it is formally defined in section 3.1.\n5. Line 87, it mentions that \"the structure f a cell remains the same when ...\". Does the structure here mean the the output of the \"f\" function mentioned in the Definition 1 above?\n6. Line 90, It says \"periodic invariance is also shown necessary for generating valid crystal representation\". Why?\n7. In table 2 the paper compares the time complexity of the proposed method with ALIGNN. It would be great if some complexity analysis is provided here.\n8. Some relevant works [4, 5] regarding the periodic graphs generation can be discussed in the related works section.\n9. The code is not published for now to reproduce the results.\n\n\n[1] Xu, Minkai, et al. \"Geodiff: A geometric diffusion model for molecular conformation generation.\" arXiv preprint arXiv:2203.02923 (2022).\n[2] Kim, Seohyun, Jaeyoo Park, and Bohyung Han. \"Rotation-invariant local-to-global representation learning for 3d point cloud.\" Advances in Neural Information Processing Systems 33 (2020): 8174-8185.\n[3] Court, Callum J., et al. \"3-D inorganic crystal structure generation and property prediction via representation learning.\" Journal of chemical information and modeling 60.10 (2020): 4518-4535.\n[4] Xie, Tian, et al. \"Crystal diffusion variational autoencoder for periodic material generation.\" arXiv preprint arXiv:2110.06197 (2021).\n[5] Wang, Shiyu, Xiaojie Guo, and Liang Zhao. \"Deep Generative Model for Periodic Graphs.\" arXiv preprint arXiv:2201.11932 (2022).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "84WLKF3wUbi", "reviewer": "Reviewer_VsGM", "summary": " The paper studies representation learning for periodic objects such as crystal. The paper formally defines the periodic invariance property and periodic pattern encoding. To achieve both periodic invariance and periodic pattern encoding, the author proposes two type of constructions of local graphs, modified from existing methods (CGCNN and Graphnormer). The author further develops a proper message passing based transformer over the proposed graph construction method. Overall the paper is well-written and informative for representation learning on periodic objects. The experimental result shows the effectiveness of their designs.  ", "questions": "Please see previous comments. ", "limitations": "NA", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The written is extremely clear and informative in terms of analysis and literature review. Reading the paper is a kind of joy. The author also clearly state the difference with respect to several similar previous works.\n2. The proposed method achieves all invariant properties desired. The design is simple, clean, and meaningful. \n3. The proposed properties for periodic objects are formal and informative. if these properties are proposed the first time here, this is a good contribution. \n\nWeaknesses:\n1. Even stated several times, the proposed graph construction methods are similar to previous works. In this perspective, the novelty is a bit reduced. \n2. The performance improvement comparing with the strong baseline ALIGNN is small. Given ALIGNN uses angular information, it would interesting if the author can adapt angular information to the designed Matformer. If the author can further improve the current Matformer, this paper will become extremely good. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "4_dMLdjd7SO", "reviewer": "Reviewer_AKt8", "summary": "This paper considers the property prediction tasks for crystal materials. This problem can be viewed as predicting a global target for a graph. The uniqueness is that the graph is periodic. The graph crystal structure repeats in the 3D space. Past works didn't really emphasize on this periodic invariance, or handle the periodic pattern encoding. The most commonly seen strategy is to set a fixed radius for an atom, and top $t$ closest neighbor atoms to this atom are connected to this atom with an edge. Such design implicitly considers the periodicity of the crytals. However, in some corner cases, such invariance may not be guaranteed. To this end, this paper proposes to explicitly capture the periodicity through self-connecting edges. Intuitively, the self-connecting edges connect one atom with its nearby duplicates. Since it is in the 3D space, there are multiple duplicates in different directions. So, multiple self-connecting edges are added to model the periodicity. The resulting model, Matformer, outperforms other GNN-based methods on two datasets, materials project and JARVIS. ", "questions": "1. I belielive $\\ell_1, \\ell_2, \\ell_3$ are 3d vectors with only one dim non-zero?\n2. It's better to include $\\ell$ as a superscript in your denoting $f^*_i$ for the $\\ell$-th layer\n3. What do you store in edge feature vector?\n4. In Table 1, it seems that the numbers from your reproduction and the original paper differ by a lot. Why is that?\n5. Why GATGNN etc. do not show up in Table 3?\n6. Whether to add the angle information is always debatable in this field. ALIGNN often becomes the second best numbers and it encodes angles. But you claimed angles \"induce high complexity\", could you clarify on this?\n7. For each material property, do you train a separate model? or you use one learnt representaion for all?", "limitations": "1. I do believe it would be better for authors to include the discussion for the comparison between Euclidean neural nets and graph neural nets in encoding the crystals.\n2. Some interpretations on the results can help me to better understand the perforamnce gain.\n3. More evaluation metrics would make it more convincing. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n1. The paper is written clearly. Though there are some typos, the general idea and methods are easy to follow. And the real-world problem is also important in materials science.\n2. The ablation study is presented. I think the key argument of this paper is the periodic invariance. So proving the importance of the periodic module is critical.\n3. Multiple datasets are included for validation. \n\nCons:\n1. Essentially, the paper extends the existing graph based methods for crystal property prediction. However, another direction is Euclidean neural networks. These models are not compared in this paper, such as E3NN.\n2. The authors decide to drop the angle information due to high complexity. But ablation study should be provided to support this claim.\n3. As I mentioned, the ablation study is critical for performance and operations. It would be better to expand the ablation study section and move the dataset descriptions to appendix.\n4. From the comparison numbers, the improvements over others like ALIGNN are not that significant. \n5. Experiments only evaluate MAE. More metrics can be added.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept