Rethinking the role of frames for SE(3)-invariant crystal structure modeling
Authors: Yusei Ito
Tatsunori Taniai
Ryo Igarashi
Yoshitaka Ushiku
Kanta Ono
Venue: ICLR 2025 Poster (2025)
Keywords: Materials Science
Invariant Networks
Transformer
Physics-Informed ML
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: Propose dynamic atom-wise frames aligned with attention-based interatomic interactions on the structure, rather than with the structure itself.

Abstract: Crystal structure modeling with graph neural networks is essential for various applications in materials informatics, and capturing SE(3)-invariant geometric features is a fundamental requirement for these networks. A straightforward approach is to model with orientation-standardized structures through structure-aligned coordinate systems, or  “frames.” However, unlike molecules, determining frames for crystal structures is challenging due to their infinite and highly symmetric nature. In particular, existing methods rely on a statically fixed frame for each structure, determined solely by its structural information, regardless of the task under consideration. Here, we rethink the role of frames, *questioning whether such simplistic alignment with the structure is sufficient*, and propose the concept of *dynamic frames*. While accommodating the infinite and symmetric nature of crystals, these frames provide each atom with a dynamic view of its local environment, focusing on actively interacting atoms. We demonstrate this concept by utilizing the attention mechanism in a recent transformer-based crystal encoder, resulting in a new architecture called **CrystalFramer**. Extensive experiments show that CrystalFramer outperforms conventional frames and existing crystal encoders in various crystal property prediction tasks.

Published as a conference paper at ICLR 2025
RETHINKING THE ROLE OF FRAMES FOR SE(3)- INVARIANT CRYSTAL STRUCTURE MODELING
Yusei It,, Tatsunori Taniai'! Ryo Igarashi', Yoshitaka Ushiku', and Kanta Ono? "Contributed equally. !OMRON SINIC Corporation "Osaka University https: /tops:///mron--. sinicx, github to/ coocrrstaalfammer
ABSTRACT
Crystal structure modeling with graph neural networks is essential for various applications in materials informatics, and capturing SE(3)-invariant geometric features is a fundamental requirement for these networks. A straightforward approach is to model with aligned coordinate systems, or "frames." However, unlike molecules, determining frames for crystal structures challenging due to their infinite and highly symmetric nature. In particular, existing methods rely structure, determined solely by its structural information, regardless of the task under consideration. Here, we rethink the role of frames, questioning whether such simplistic alignment with the structure is sufficient, and propose the concept of dynamic frames. While accommodating the infinite and symmetric nature of crystals, these frames provide each atom with a dynamic view of its local environment, focusing on actively interacting atoms. We demonstrate this concept by utilizing the attention mechanism in recent transformer-base crystal encoder, resulting a new architecture called CrystalFramer. Extensive experiments show that CrystalFramer outperforms conventional frames and existing crystal encoders in various crystal property prediction tasks.
structures through structure-
statically fixed frame for each
INTRODUCTION
Geometric graph neural networks (Xie & Grossman 2018 Chen et al., 2019 Choudhary & DeCost
202T) et al. 2023). including transformer variants (Yan et al. 2022; 2024; Taniai al. (002)),
play central role machine learning (ML)-based structural modeling of materials. This technology offers a powerful alternative to conventional simulation methods, such as density functional theory (DFT) calculations, enabling high-throughput prediction of material properties. Furthermore, it
serves as the basis for various ML applications in materials science, such as material embedding
learning (Suzuki et al. 2022 2025 Chiba al. 2023) and crystal generation (Jiao et al.) 2023).
A key requirement for these networks is the ability to capture essential features of materials embedded in their crystal structures. Crystal structures are periodic, infinitely repeating arrangements of atoms in 3D space, typically represented by minimum repeatable patterns called unit cells. Material properties, such formation energy and bandgap. are invariant under rigid transformations (i.e., rotations and translations) in crystal structures, as well as under variations in their unit cells. This fact leads to the so-called periodic SE(3) invariance Yaan al. 2022) as an essential property for crystal encoders. Therefore, recent studies have explored various forms of richer yet invariant structural information beyond the simplest interatomic distances (Chen & Ong 2022 Duval et al. 2023 Yan et al. 2024).
One approach that has shown promising results for molecules (Puny et al. 2022 is the use of "frames." " A frame is coordinate system aligned equivariantly to a given structure to provide an ceintaatioo-ttandaddieee view of the structure (see Fig. [ left). Frames allow arbitrary networks to directly exploit rich 3D structural features, including the relative positions between atoms and their directions, without imposing any architectural constraints. However, determining frames for crystals is more challenging than for molecules, primarily due to the infinite and symmetric nature of crystals.
In this work, study new family of frames for crystal structures in rethinking the role of frames. We hypothesize that the essential role of frames is not merely to provide a structure structure-aligned coordinate system for given structure, but rather align the coordinate system with the interatomic interactions

---

Published as a conference paper at ICLR 2025
Conventional static frame
Proposed dynamic frames
Frame
Layer
Layer 2
Figure I: Conventional static frame and proposed dynamic frames. Conventional frames are determined statically to align with the structure, ensuring consistency under rotation and providing canonical global representation of the structure. This consistency is schematically illustrated by the curved arrows. By contrast, the proposed dynamic frames are determined for each atom in each message-passing layer. by considering the local dynamic environment around that atom in that layer.
acting on the structure. Following this belief, we propose a novel concept of dynamic frames. These frames define local coordinate systems centered individual atoms by dynamically accounting for the atoms actively engaged in learned interactions in each interatomic message- paasssgggg layer (Fig. [] right). This concept challenges the conventional notion of 'static frames,' which are based on the premise of providing fixed views of structures (Puny et al. (022).. Thus, whether such dynamic frame is effective or not is an unexplored non-trivial question, which we aim to answer.
To verify this concept, we develop several types of dynamic frames by utilizing the self-attention mechanism (Taniai et al., (024) quantify the interaction engagement. We conduct extensive comparisons on datasets derived from the JARVIS, Materials Project (MP), and Open Quantum Materials Database (OQMD). Our results show that our method outperforms existing frame methods for crystals (Duval et al. 2023 Yan et al. (2024) and other state-of saate-of-the-eeee art networks (Choudhary &
DeCosi Chenn & Ong 2022 Yan et al. 2022 2024, Lin et al., 2023 Taniai al. (0244) across various crystal property prediction tasks. We release our code online.
2 PRELIMINARIES
2. CRYSTAL STRUCTURE
crystal structure is described by its 3D unit cell slice, denoted as (A, P, following Yan et al. (2022). A unit cell is parallclepipedal structure containing finite number, say N, of atoms. The species (atomic numbers) and 3D Cartesian coordinates these atoms are provided as = (a1, a2, aN] NIxN and P [p.. P2. PN] R The parallelepipedal cell shape is given by three vectors: L = 14,, 2,, La] E 3x33 called lattice vectors. By tiling the parallelepiped unit cell fill 3D space, the species and positions of all the atoms in the crystal structure are determined as
Â (()())aan))) = a,, E 23 SiiNN,
P = (Pi(m) Pi(n)) p. Ln,n E 1i,1
(2)
Following Taniai et al. duplicate by a unit-cell translation: In = nili + nzl2 + nzl3. We use and j(n) similarly.
2024)), we use i to denote the i-th atom in unit cell, and i(n) to denote its
2.2 TRANSFORMERS FOR CRYSTAL STRUCTURES
Geometric graph neural networks are used crystal encoders in various materials-related tasks. These encoders typically represent the state of given crystal structure by set of atom-wise abstract state features, X 1 222..... ...; xy E RdxN These states initially provided as atom embeddings, (0) AtomEmbedding(A4 (A), which only symbolically represent atomic species. The encoders then evolve these states through interatomic message-passinng layers, ((++1) +- f'(((1), P, T.). to
2

---

Published as a conference paper at ICLR 2025
eventually reflect the atomic states in the given structure appropriate for target task. Since the seminal work of Xie & Grossman (2018) and |Schütt et al. (2018). graph neural networks (GNNs) have long been the standard for crystal encoders until the advent of transformer- measdermer-b networks by recent work Yan et al.
2022
2024 Taniai et al. 2024).
In particular, Taniai et al. (2024) have developed simple physics-informed formalism for crystal encoders using a self-attention mechanism. By imitating interatomic potential summations for energy calculations in physics simulations, they model the evolution of current state using infinitely connected distance-decay attention. This attention mechanism models the interactions between each unit-cell atom i and all the infinitely repeating atoms j(n) in the entire crystal structure as
9: ky vdk
Pa(m) - Pall?) (1o, + West())) 207
exp
(3)
Z
Here. query q, key k. and value v are linear projections of current state æ. Scalar oy is tail-length variable for Gaussian distance-decay attention, adaptively derived from C,. Vector Piy(n) is geometric position embedding that encodes the distance, |[)(()) pill. between atoms i and j(n).
Scalar Z; = E, En exp(q? ( //jv - (l))(n) pill² /2//2² ) is the normalizer of softmax attention
weights. The exponential distance-decay factor in Eq. 3 provably ensures its rapid convergence within
a finite range of unit-cell shifts n (Taniai et al. 2024).
Their method, called Crystalformer, enjoys good balance between strong physically mottivate inductive bias and the flexibility of abstract feature representations. It considered the state of the art with other GNN-based (Lin et al. 2023) and transformer-based (Yan et al., 2024) methods.
We utilize Crystalformer as baseline in this work. This because its architecture closely follows the standard softmax attention (Vaswani et al. (2017) and is suitable to demonstrate our concept of dynamic frames, while other existing transformers ( (Yan et al. 2022 2024) use distinct channel-wise sigmoid attention. We discuss this more in Sec. Our method, described in Sec. 3 extends position embedding vij(() in Eq. 3 incorporate richer, invariant information beyond distances [|p)(m) - Pill-
2.3 FRAMES FOR SE(3)-INVARIAN) STRUCTURAL MODELING
Frame averaging. Puny et al. (2022) have introduced Frame Averaging (FA) as a general framework to adapt networks be invariant (or equivariant) to certain symmetries in input data. Although FA is originally grounded in group representation theory, we provide a high-level review focused on SE(3) 3E(3)-invariant modeling of 3D point clouds. Given point cloud P. FA computes frame, F E F(P), as a coordinate system inherent to and aligned with P (Fig. [ left). For instance, F is principal component analysis (PCA) applied to P. Each frame F' thus detines geometric transformation that maps P to canonical, votation-invarial representation, FP. However, F(P) may not uniquely provide a single frame due algorithmic ambiguities in F or symmetries in P. Even in such cases, FA allows us to derive rotation invariant (i.e., (O(33)invariia)) networks for from any given networks f, by averaging f's outputs over all possible finite frames, as follows:
fx(X
Σ F(X,FP). FEF(P)
(4)
[F(P]
The translation invariance is further attained by formulating f with relative positions (e.g., Fp,) bringing SE(3) invariance to fy. FA can powerfully adapt arbitrary networks to be SE(3) invariant without constraining the architectural design. However, hinders efficiency, as the computation increases with the number of possible frames. Stochastic FA by Duval et al. (2023) mitigates this issue by randomly selecting single frame from F(P) during training. This scheme enforces networks f to learn the invariance to frame variations, approximately achieving SE(3) invariance.
PCA frames. Puny et al. (022) originally applied FA for molecules using PCA-based frames, and Duval et al. (2023) later extended it for crystals by simply treating unit cell structures P as finite-sized point clouds. These PCA frames compute three orthogonal eigenvectors {e1,e2, e2, es} of the covariance matrix for P. corresponding to eigenvalues X₁ X 2 Àg, as the frame axes: F = [e,, E2, ez]T. Because of the sign ambiguity of the eigenvectors, PCA produces eight frames for 0(3)E(3) invariance and four frames for SO(3)SS((3 invariance with the restriction of det(F)
3

---

Published as a conference paper at ICLR 2025
Although PCA is well-established, it suffers from eigenvalue degeneration for highly symmetric data, such as crystal structures. For example, PCA for cubes produces identity covariance matrices up to a arbitrary vectors E R The crystal frame construction by Duval et al. (2023) is thus vulnerable to this degeneration issue and, moreover, sensitive to unit-cell variations of the same crystal structure.
constant scale, whose eigenvectors
Lattice frames. Yan et al. (0224) have proposed frames based on the lattice vectors of crystals, as similar reduced cells (i.e., uniquely determined minimum cells) (Ngglli 1928). Specifically, their method selects lattice point, e = 2 + n222 + ngL3, with the minimum non-zero norm ||e||2 as first axis e1, and selects the second and third smallest ones as axes e₂ and es while ensuring rank(e1, e2, e3) is full. The signs of these axes are adjusted so that the angles between e₁ and e2 and between e₁ and ez become acute and the coordinate system is fight-handed (i.e., det (F) 0).
Notice that these existing frame methods for crystals, specifically PCA and lattice frames, all provide a statically fixed frame for each crystal structure. Also, both rely on unit cell representations (either points P or lattice vectors L). which are rather artificiall-intrrodueed crystal descriptions that may not necessarily reflect the physical properties of materials (see Appendix [ for more discussion). These observations motivate us to propose the concept of dynamic frames, as we discuss next.
DYNAMIC FRAMES
In the search for effective frames for crystals, we challenge the conventional notion of frames, which implicitly follows the simple premise of representing structures in a canonical manner (Puny et al. 2022] Duval et al. 2023 Yan al.] 2024). Let us reconsider how frames work in GNNs, whose interatomic message-passing layers are assumed to include the following general operation:
x Σ
Σ (is(t)) P).
This equation describes that state x; of each unit-cell atom i is evolved through abstract influences or messages, fiejj(n), from atoms j(n) in the crystal structure, with scaling weights Wej(n)- In standard GNNs (Xie & Grossman] 2018), these weights are pre-defined as neighborhood graphs with a cut-off
radius. In recent transformer architectures, the weights determined dynamically via self-attention, with (Yan et al. 2022 2024) or without (Taniai et al.) 2024) relying on an explicit cut-off radius.
The role of frames in Eq. is offer, for the design of fit-j(m). more informative invariant edge features than distances through frame-projected coordinates FP. From this perspective, constructing shared frame for the state updates of all atoms i, as done in conventional methods, not preferable. This is because the frame construction can be influenced even by atoms j(n) with zero weights in 5 other words, when updating the state of atom i in Eq. []] this atom has its own partial and local view of the entire crystal structure, P. where weights Way(n) act as a mask on the structure.
This interpretation leads to the concept of dynamic frammec That is, we define frames locally for each atom i align with its interatomic interactions dynamically acting on the structure, rather than directly aligning with the structure itself. We denote these dynamic atom-wise wise frames as F. Each F, is determined based on the masked view of structure P with weights Wij(n), by emphasizing or de-emphasizing the presence of atoms (n) with larger or smaller weights. Thus, these frames F; change dynamically depending on target atoms and also on the layers in a GNN, as shown in Fig. []
We hypothesize that dynamically adapting frames for each atom i in each message-passing layer (Eq. 5) provides better invariant edge features via projected coordinates F.P. We also point out that these frames are defined with the entire crystal structure, Р. reconstructed from (P, L). This fact highlights an advantage of our frames being invariant to unit cell variations within the same structure.
3.1 FRAME DEFINITIONS
We now present several instances of this new family of frames. These frames F; are constructed for each target atom in each message-passing layer (Eq. 5). by using coordinates P and weights (i)()) of atoms j(n) in the structure. We typically assume ((j(n)) 0, but use real-valued
4

---

Published as a conference paper at ICLR 2025
weights, for example, by using their absolute values for frame construction. For brevity, we denote rij(m)) ()))))) Pillz and Fi)(m)) (P)()) -Pi)/Tas(m)> both derived from P.
Weighted PCA frames. The first instance of dynamic frames extends the original PCA frames (Puny
et al.. 2022 Duval et al., (023). For each target atom i in each message- passiig layer, compute
3 x weighted covariance matrix, Σ; = E, 2n Waj(n) Tij(m) F (j()) and computes its orthogonal
eigenvectors { e2; ea} as the frame axes: F: = [ei, e2, ea]² For the sign ambiguity of eigenvec- tors, we adopt the stochastic FA (Duval et al. 2023) and generate a single frame by randomly flipping the signs of these vectors while ensuring (c) = 1. However, there remains another possible ambiguity in this weighted PCA scheme owing to eigenvalue degeneration by symmetried"
Max frames. To avoid the degeneration of PCA, we also propose directly selecting atoms j(n) with large weights Wij(n) and using their directions (()(n) to determine axes {ei, ea) of F;. Specifically, the first axis, E1, is selected as Tij(m) with the maximum weight Wij(n)- For the second axis, we find Pij(m] with the maximum dajuusted-weight (1- le. Pij(m)) ] (n))))) avoiding a direction parallel to e1. The selected vector, denoted as F22, is further orthogonalized using the Gram-Schmile method as é2 + '2 - (e) F2)ee, and normalized to a unit vector as e2 + é2/ @2/|êe]|o Finally, the third axis is simply obtained as e3 this process, multiple atoms may have the same weight. For this ambiguity, we add small perturbation noise to each weight Wij(n), resulting in randomly selecting single frame from possible ones. This perturbation scheme is considered a type of stochastic FA (Duval et al. 2023) outlined in Sec. 2.3
X e2, which ensures the orthogonality and det = 1. In
Since these frame construction processes are not stably differentiable, we omit the computation of the gradients from frames F; to weights Wij(n) during training" Still, weights Wij(m) receive gradients from x' in Eq. to learn their main function: allowing or blocking messages fiej(n) from j(n) to i. Therefore, can train a network with dynamic frame construction without using frame gradients.
3.2 CRYSTALFRAMER ARCHITECTURE
We demonstrate the proposed concept using Crystalformer (Taniai et al. (024) as the baseline, as mentioned in Sec. 2.2 and develop a new architecture, CrystalFramer (Fig.
We here regard Eq. 3 as Eq. 5 Thus, regard the softmax self-attention weights (i.e., exponential weights normalized by Z. in Eq. 3) as dynamic scaling weights Waj(m) in each message-passing layer (Eq. 55. Likewise, we regard the position-augmented value vectors, Vj Pisj(n)) . as messages fie-j(n)- During the state update of each atom x; using Eq. 3 we first compute the attention weights as Wey(n)- Then, dynamically construct a local frame each atom i as matrix F,, by following one of the procedures outlined in Sec. 3.1 Finally, we compute (bij(n) using F; and perform Eq. Below, we explain how derive invariant edge features (bsj(n) given that frame F. is determined.
Invariant edge features using a dynamic frame. For invariant edge feature t)i(()). Crystal- former originally uses linearly projected Gaussian basis functions (GBFs) to encode distance Tiy(n)- Specifically, GBFs are defined as a mapping from a scalar to a vector of pre-defined dimension D, b(x) [b1, bg. ,bb"", whose k-th component is computed as a Gaussian given by
bx(2; Pk: σk) = exp (-(x - /20
(6)
Here, (4 and 0% are pre-defined as μık Mmnin + (k: (0-1))(e () /mmin) //min)/ 1 -D) and s(Mmux /min)/ (DD - 1), where {Pmax: Mmmnn 8, D) are hyperparameters. Intuitively, b(x) encodes scalar x into soft one-hot vector using D Gaussians uniformly distributed between Mmin and jmax. The widths of these Gaussians, Okk are proportional to the interval distance, controlled by scaling factor s.
'We confirmed that covariance matrices Σ, computed with pretrained Crystalformer model suffered from eigenvalue degeneration at two degrees in about 10% of cases and three degrees in about 1% of cases. These cases cause rotation ambiguities for two or three (all) axes F,. To mitigate this issue, we add small perturbation noise to Wij(m) in >, which stochastically breaks the symmetries in the structural data and empirically helps to compute non degenerate cigenvalues and eigenvectors. This scheme is considered type of stochastic FA.
"The gradients of the eigenvectors in PCA become numerically unstable when the eigenvalues are degenerate, as the gradients depend on the computation of 1/(A; - 2ς) for + j. Also, the max- fax-frame procedure is not differentiable due to the use of argmax operations. Although we tried approximating the gradients of argmax, for example, by using straight-through estimator technique or temperature annealing of softmax, simply ignoring the frame gradients gave the best results.
5

---

Published as a conference paper at ICLR 2025
Output
X' = { x/2
x'y
Feed forward
Pooling
fi-j() Dj + vij(m))
E, En Wistofi-j(0n
Invariant edge features
X 4
iijj(n)
Feed forward
Wij(n)
Frame construction
Softmax
Distance-decay attention factor
6 6(3) Vitoo 1/on)- (Kay ((n))
Linear
|P/60 pill 20
Concat
Scaled dot product
kj
Multi-head
attention
Linear
Linear
Linear
Crystal atructure
X1
X22
XN
Spedas info (000000))
X
Figure 2: CrystalFrames architecture. Dynamic frame construction and Srume-based invariant edge features (highlighted in red) are introduced to transformer for crystals (Taniai et al. 2024).
We retain their distance-basee edge feature and further add frame-based edge features to ()))(n))
Specifically, following existing work (aan et al. 2024), we represent direction vector Fij(n) invariantly by projecting it onto the frame coordmate system, calculated as C₁ (n)( the cosine the angle between k-th frame axis e₁ and direction Fi)(n)-
0z)(m) F)Fij(va). Its k-th component is
Each component is then converted to vector using GBFs. By combining the setaace-bbsse and three angle-based features via linear projections, we obtain our geometric relative position encoding:
[i)))) Wobass (ras(m))
Σ k=1,2,3
Wibbaggl
(o(A) (0(())
(7)
This ())((n) as whole essentially encodes the 3D relative position vector: T(y(n) P(()) P Furthermore, the angle components of (l)()) can be interpreted as encoding the deviations (m)(m) from the three primary directions of interatomic interactions (i.e., G1, 62 e3) around target atom i. Here, four weight matrices (Wo, W1, W2. Wa} are trainable parameters provided per layer. We also use two types of GBFs, baist and b anggt with different hyperparameters for the distance and angles. Specifically,,
{Aminy Mmax; 8, D} { 64 A, 14.0A, 140 for bass suggested by Taniai et al. (2024). We also set them to {-1.0, 1.0,4.0, 64} for bangl, using the cosine range [-1.0, 1.0 and relatively larger width-scale s. which empirically works better for angles. If Fis(m) is undefined 0.
due to division by zero (i.e,, when j(n i), set bangl
Overall architecture. The proposed network precisely follows the Crystalformer architecture (Taniai al. 2024) as shown Fig. with the addition of frame construction (Sec. 3.1) and angular edge features (Eq. 7 highlighted in the figure. As we will see in Sec. 5 these simple extensions lead to drastic performance improvements over the baseline. Below, we summarize the key design aspects of the network. The overall architecture consists of an input atom- atom-embeddding layer, stack of four self-attention blocks, global mean pooling, and a final feed-forward network with two linear layers. The self-attention blocks adopt a normalization-fies architecture (left part of Fig. 2 by Huang et al. (2020) for enhancing training stability. The infinite summation in self-attention (Eq. 3 is computed convergently and efficiently. Specifically, sufficiently cover the neighbor radius of 3.5 based on dynamic Gaussian tail-length o,. We also employ multi-head self-ate attentttntio in the original transformer (Vaswaan al.) (0177), with eight heads. Frames are constructed per unit-cell atom, per head, and per layer. For further architectural details, please refer the original work (Taniai al. 2024).
adaptively determine the range of unit-cell shifts to

---

Published as a conference paper at ICLR 2025
4 RELATED WORK
The notion of invariant structural modeling encompasses various invariance properties. The most basic one is invariance the permutation of data-point indices in structural data (i.e., A and P in Sec. 2. It was first addressed by PointNet (Qi et al. 2017) and DeepSets (Zaheer et al.) 2017) and is now inherited by GNNs and transformers. The ML community has subsequently focused on invariance under geometric transformations, such rotations with or without translations (i.e., SO(3)0(3) or SE(3)E(3) invariance). In particular, the periodicity of crystals introduces more complex invariance notions, such periodic SE(3) invariance (Yan al., 2022). requiring invariance to unit-cell variations within the same crystal structure. These geometric invariance properties have been explored in three primary approaches: 1) invariant features, 2) equivariant features, and 3) frames. We briefly review these approaches, focusing primarily on crystal structures.
Invariant features.
The most straightforward approach is to rely entirely on inherently invariant geometric quantities, such as the lengths of relative position vectors, throughout a model (Xie &
Grossman 2018 Chen et al. 2019 Ying al. 2021: Yan et al. 2022 Taniai et al. 2024). However, such distance-based GNNs and transformers have limited expressibility (Pozdnyakov & Cerionii 2022). Thus, recent studies have explored more advanced geometric features, such as the angles between triplets in 3-body interactions (Park& & Wolverton 2020) Choudhary & DeCost [202]] Chen Ong. 2022), though the cost of increased computational complexity. More recently, PotNet (Lin et al. 2023) employed the periodic summation of pre- pre-defined interatomic scalar potentials as more physically informed invariant edge features, compared to distances.
Equivariant features. Equivariant networks, grounded in group representation theory, form another active research area in 3D structural modeling and include invariant networks as special cases. While we refer readers to recent surveys (Gerken et al.) [023]] Duval et al.) 2024 Han et al. 2024) for more comprehensive reviews, the initial approach using GNNs for 3D point clouds and atomic systems was proposed by Thomas et al. (0118). Subsequently, this approach has been extended, for example, to introduce improved nonlinearities (Batzner et al., 2022 Brandstetter et al. 2022), attention mecha- nisms (Fuchs al.] 2020), greater efficiency (Liao & Smidt 2203 Liao 2022)) in molecular structure modeling. Essentially, these methods use spherical harmonic representations of unit di- rection vectors Pıj as rotation-equivariann edge features, which are then equivariantly transformed through specially designed networks. These equivariant features form type-L vectors in a pyramidal shape, where type-0 features encode invariant scalars and type- features represent 3D equivariant vectors, such as forces. However, these networks are constrained by limited nonlincarity forms and the increasing computational complexity required to incorporate higher- frequenyy components in higher degrees L. Due to these limitations, their application in crystals is more limited compared to molecules. For example, eComFormer (Yan et al. 2024) has exploited equivariant features in part within each message-passing block for invariant crystal property prediction.
Frames. As explained Sec. 2.3 Puny et al. (2022) introduced FA and applied PCA frames for molecules. Duval et al. (2023) further extended this work in two ways: by proposing stochastic FA to improve efficiency and by applying PCA frames to crystals by treating their unit-cell structures P as finite point clouds. Cheng et al. (2021) used plane waves in crystal structures invariant positional features, implicitly employing reciprocal lattice vectors as a frame. Similarly, Yan et al.] (2024) proposed iComFormer, using transformed lattice vectors with reduced ambiguities as a frame. Lin et al. 2024) proposed minimal FA, an efficient FA method ensuring exact invariance and equivariance.
Our work contributes to this line of research frame-based invariant networks, providing new perspective on the previous notion of frames through the introduction of dynamic frames. While several local frame methods exist in the molecular modeling literature (Du et al. 2022; 2023 Pozdnyakov & Ceriotti 2023), they do not incorporate dynamic frames as we do (see Appendix B for a comparative discussion). We integrate these dynamic frames into simple distance-based transformer model for crystals (Taniai et al., 2024) to boost its expressive power.
5 EXPERIMENTS
To validate the effectiveness of the proposed dynamic frames, we conducted extensive experiments on crystal property prediction. We compared our method with conventional PCA frames (Duval et al. 2023). lattice frames (Yan et al. 2024), and other state-of-the-art architectures for erystals.
7

---

Published as a conference paper at ICLR 2025
Datasets. We use three datasets: JARVIS (55,723 materials), MP (69,239 materials), and OQMD (817,636 materials), using snapshots available through a Python package (jarvis-tolls. These datasets provide several material properties, such formation energy and bandgap, simulated by DFT calculations. For further dataset descriptions, see Appendix [ Choudhary & DeCost (202T)
and Yan et al. (2022) evaluated many methods on the JARVIS and MP datasets using consistent data splits. Following these and later studies (Lin et al., 20233] Yan et al.) [0224] Taniai et al.) (024), we use the same data splits and cite their reported scores to reduce computational burden. Unlike these studies, we also use the much larger-scale OQMD dataset to assess scalability.
Training settings. To assess the pure effects of introducing the frames. we precisely follow the train- ing settings of the baseline method, Crystalformer (Taniai et al., [024]. with only one modification. We have increased the number of training epochs to account for the increased complexity of our edge feature design (i.e., our method takes longer to converge, but reduces validation losses more rapidly). Specifically, for the JARVIS dataset, we train our model from scratch by optimizing the mean absolute loss function using Adam (Kingma & Bal 2015) for total of 2000 epochs, while enabling the frames from the beginning. A summary of detailed training settings, including the number of epochs, batch size, and learning rate for the three datasets, be found in Appendix D
5.1 CRYSTAL PROPERTY PREDICTION
Tables Gooles[andd aand 2 extensivvely compare the mean absolute errors of the proposed and existing methods for the JARVIS (5 tasks) and MP (4 tasks) datasets. Several earlier methods are omitted from the tables and provided in Appendix E Overall, our method with max frames outperforms others in most tasks, significantly enhancing the performance of the baseline Crystalformer model. Such improvements never fade even when feeding the much larger OQMD dataset, as shown in Table is important to note that the current stat-oof-the-an ComFormer, uses finely-tuned hyperparameters (e.g., learning rate, loss functionn number of layers, graph structure) for each individual task, whereas we simply adjust the number of epochs and batch size for each dataset. In the bottom parts of Tables [] 2 our weighted PCA frame method shows relatively limited improvements (see Appendix [F for detailed discussion). Nevertheless, it outperforms its conventional counterpart using PCA frames. Additionally, we evaluated a variant using static local frames. These frames are similar to max frames but constructed with static weights, Way(n)) = exp ())()))) As result, these static local frames rely only on the distances to neighbors and do not account for dynamic self-attention weights. The max frame method outperforms this static counterpart in most tasks. These results successfully validate the effecttivness of our concept of dynamic frames.
5.2 EFFICIENCY COMPARISON
Table 4 compares the model efficiencies of several top-performing architectures. Notably, despite the superior performance of the proposed method, it requires significantly fewer parameters than PotNet, Matformer, and iComFormer. Compared to Crystalformer, our method introduces only a small overhead of about 100K parameters owing to projection matrices {Wi, W2, W3} in Eq. 7 Given the performance gains shown in Tables [T-]] this high cost cost-performance ratio also highlights the effectiveness of our dynamic frame feature design. In terms of runtime, the test time is faster than PotNet, Matformer, and iComFormer, which are hindered by relatively heavy data preprocessing. However, compared to Crystalformer, the training and test times are more than double, mainly due to the increased computation cost of En. Wij(m) (n)(n)) As noted by Tanial et al. (2024), this term is the primary bottleneck in Crystalformer and also in our model. In Appendix G Tables A2 A55), we show that a lightweight configuration for GBFs reduces the training time by approximately 42% while maintaining comparable accuracy. Pre-training without frames to efficiently learn attention weights first may also accelcrate training. In Appendix H we further discuss scalability for large structures.
DISCUSSION AND LIMITATIONS
Visual analysis. Figure 3 displays four types of frames generated for a test material TVASP-30609 in the JARVIS formation energy prediction task. While PCA (Duval et al.) 2023) and lattice (Yan) jet al. 2024) frames are static, the proposed weighted PCA and max frames exhibit dynamic behavior based learned attention weights. In each layer, our frames capture distinct local motifs, such as
8

---

Published as a conference paper at ICLR 2025
Table Property prediction results on the JARVIS dataset. Accuracies are mean absolute error. The sizes of training, validation, and test splits are listed under each property name. Bold indicates the best results, underline the second best. Full results covering earlier methods are in Appendix E
Form, energy
Total energy
Bandgap (OPT)
Bandgap (MB))
E holl
44578/5572/5572
44578/5572/5572
44578 5572/5572
14537/1817/881
44296/567/5537
Method
eVittom
eVatoo
eV
eV
Matformer (Yan all
2032
0.0325
0.035
0.137
0.30
0.064
PotNet ef A123
0.0294
0.032
0.122 127
0.27
0.055
cComFormar
2024
0.0284
0.032
0.124
0.28
0,044
iComFoomer Yán cf
M1Z4
0,0272
0.0288
0.122
0.26
0.047
Crystalfomner Tanaai
2024
0,0306
0.0320
0.128
0.274
0.0463
w/ PCA frames DDiall ct al
2023
0.0325
0.0334
0. 144
0.292
0.0568
lattice frames
0.0302
0.0323
0.125
0.274
0.0531
w/ static local frania
0.0285
0.0292
0.122
0.261
0,0444
weightte PCA frames (proposed)
0.126
0.279
w/ max frames (proposed)
0.0263
0.0279
0.117
0.242
0.0471
Table 2: Property prediction results on the MP dataset.
Formation energy
Bandgap 60000 4000/4239
Balk modulus
Shear moduluss
3000/4239
4664/303/393
4664/392/395
Method
eV/stomm
eV
leg(CiPu)
leg(CiPu)
Matformer
0.021
0.211
0.043
0.073
PotNet
0.0188
0,204
0.040
0.065
oComFormer
0.0182
0,202
0.0417
0.0729
iComFormer
0.0183
0.193
0.0380
0.0637
Crystalformer
0.0186
0.198
0.0377
0.0689
w/ PCA frames
0.0197
0.217
0.0424
0.0719
w/ lattice frames
0.0194
0,212
0.0389
0.0720
w/ static local frames
0.0178
0,191
0.0354
0.0708
w/ weighted PCA frames (proposed)
0.0197
0.214
0.0423
0.0715
max frames (proposed)
0.0172
0.0338
0.0677
Table 3: Property prediction results on the OQMD dataset.
Form. energy (eViatom) 654108 /81763/ 81763
Bundgap (e V)
E bull (eVatom)
Method
653388/81673 /1673/81673
654108/81 08100881163/8/108
Crystalfommer
0.02115
0.06028
0.06759
CrystalFramer (max frames)
0.01871
0.05805
Table Efficiency comparison. Per-epoch training time includes validation, and per-material test time includes preprocessing, such as graph construction. The runtimes evaluated for the formation energy prediction in the JARVIS dataset using a single NVIDIA A6000 GPU with 48GB VRAM.
Model
Arch. type
Timelepoch
Test/mater
PPrrms
(Prams./bloct
GNN
43
1.8 M
Mationmer
Transfommer
60
20.4
2.9 M
544
iComFaree
Tranfformer
59
54.8 ms
5.0 M
855
Crystalforner
Transformer
32
6.6
853
206
CrystalFramer
Transformer
74
16.8 ms
952 K
231
octahedra with a green central magnesium atom surrounded by blue fluorine atoms, and tetrahedra with red central tin atom surrounded by blue magnesium atoms. These local structures are common and often distorted, as seen in this example. The ability to capture these local structures and measure distortions via relative positions may contribute to the high performance observed. A more detailed analysis, including comparative discussions on different frames, visualizations for different material, and an examination of frame evolution during training, is provided in Appendix F particular, the frame evolution analysis (Appendix F. shows that max frames converge faster during training due to the discrete nature of their construction. While this characteristic may contribute to the superior performance of max frames, it introduces noticeable discontinuities to the model and may limit generalization to out-of-domain data, as discussed in Appendix !!]
Baseline choice. This study adopts Crystalformer (Taniai et al. 2024 for demonstration, since its standard multi-hes head softmax attention is well-suited for dynamic frames. Other crystal trans- formers Yan et al. 2022 2024) use distinct channel-wise sigmoid attention, akin to maximally

---

Published as a conference paper at ICLR 2025
Crystal structure JMgSnFa)
Welghted PCA frames
5n
Mg
F
PCA frame
Lattice frame
Layer 1
Layer
Layer 3
Layer 4
Figure 3: Frame visualizations. Conventional PCA and lattice frames provide global coordinate system based solely on the structure. The proposed dynamic frames extract different structural information for each atom and layer using dynamic attention weights, shown as varying transparency.
multi-headed attention. Since frames and angular features are computed per atom, per head, and per layer, such channel-wise attention is not preferable. On the other hand, Crystalformer with Eq. 3 can be interpreted as the original fully-connected self-attention (Vaswani et al. (017))
x/ = 2, Ej exp(q, / Vdx + diy) + v. with two straightforward extensions: 1) relative position representations via diy and tbi (Shaw al. 2018) and 2) duplication of each atom j(n) using En for crystal periodicity. Given the practicability and versatility of the original transformer architecture across diverse domains (Lin et al. 2022), our demonstration serves as foundation for transformer-baue crystal encoders using dynamic frames.
Equivariant prediction. While this study focuses on SE(3) invariance, Puny (2022) extended FA to predict equivariant quantities, such as force vectors, by applying inverse mapping F to f(X, FP) Eq. prior to averaging. In our case, one potential equivariant extension would thus invariantly output atom-wise geometric quantities u, from x² (e.g., via u, W Wx') and subsequently apply the inverse mapping u, Another approach, similar to recent work (Shi et al. 2023). could tie the outputs equivariantly to the input structure, for instance, by computing un 2g Σn (ij(n) These equivariant extensions would enable predictions of forces and relaxed structures, which are crucial for surface property analysis (Chanussot et al., [021 Tran et al. (023) Bihani et al. (024). Detailed investigations into these equivariant extensions are left for future work.
Application to molecules. Transformers for molecular structures have been developed Ying et al. 202T; Wang et al. 2023, Shi al. 2023 Liu et al.) 2024), and our dynamic frames could potentially be applied to them. However, crystal and molecular structures have very different characteristics. In particular, molecular structures often contain so few atoms that they tend to form low-dimensional local structures, which may hinder the construction of effective frames. Extending methodology to molecules represents another intriguing future direction for this research.
7 CONCLUSIONS
This study revisited the challenge of determining effective frames for the SE(3)-invariant modeling of crystal structures. We hypothesized that frames should reflect the local dynamic environment around each atom, rather than the static global structure. We herein introduced the concept of dynamic frames, which leverage the strengths of interatomic interactions. These frames were integrated into an existing transformer-baseee network for crystal property prediction (Taniai et al. 2024), resulting in the Crys- talFramer architecture. Its performance was benchmarked against conventional frame construction methods (Duval et al. 2023) Yan et al., 2024) and other state-of-the- ahe networks (Choudhary & DeCost 2021| Yan et al. [2022] Lin et al. 2023). Our findings validated the hypothesis, demonstrating the superior performance of the proposed approach. Although the demonstration focused crystals, interaction based dynamic frame construction holds promise for broader applications, including surface modeling, molecular structure modeling, and ML-driven simulations of particles and fluids.
baseed
10

---

Published as a conference paper at ICLR 2025
AUTHOR CONTRIBUTIONS
YI, as an intern at OMRON SINIC X, identified the potential utility of the proposed idea for erystals, implemented most of the method in the baseline code, designed and conducted the majority of the experiments, analyzed the results, drafted the initial manuscript, and created the figures. TT conceived the method, developed and discussed the core idea and philosophy with the co-authors, co-patented the method with YI and RI, explained the baseline code to YI, helped with the coding and experiments, revised the manuscript draft, and led the rebuttal and overall project. RI provided expertise in materials science and physics simulations, helped review the molccular modeling literature, and advised on the methodology, coding, and writing. YU co-led materials-relater collaborations, provided expertise in machine learning, and advised on the methodology and writing. KO co-led materials-related collaborations, provided expertise in materials science, and advised the writing.
ACKNOWLEDGMENT
This work was supported in part by JST-Mirai Program Grant Number JPMJMI21G2 and IST Moonshot R&D Program Grant Number JPMIMS2236. TT is supported in part by JSPS KAKENHI Grant Number 24K23911. 1. RI is supported part by JSPS KAKENHI Grant Number 24K23910. YU partly supported by JST-Mirai Program Grant Number PMMUM21I2 and JST Moonshot R&D Program Grant Number JPMJMS2236. TT thanks Naoya Chiba for his insightful comment on the weakness of PCA for symmetries, which provided an initial hint for this work. The authors also thank Yuta Suzuki, Seiji Aota, and the anonymous reviewers for their valuable feedback on the manuscript.
REFERENCES
Simon Batzner, Albert Musaelian, Lixin Sun, Mario Geiger, Jonathan P. Mailoa, Mordechai Korbluuhh, Nicola Molinari, Tess H. Smidt, and Boris Kozinsky. B(3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials. Nature Communications, 13(1):2453, 2022.
doi: 10. 1e..1088/4444640 10003///411477002229999999999999
Vaibhav Bihani, Sajid Mannan, Utkarsh Pratiush, Tao Du, Zhimin Chen, Santiago Miret, Matthieu Micoulaut, Morten M. Smedskjaer, Sayan Ranu, and N. M. Anoop Krishnan. EGraFFBench: evaluation of equivariant graph neural network force fields for atomistic simulations. Digital doi: 10.1039/DDDD000077
Discovery, 3:759-768, 2024.
Johannes Brandstetter, Rob Hesselink, Elise van der Pol, Erik Bekkers, and Max Welling. Geometric and Physical Quantities improve E(3) Equivariant Message Passing. In The Tenth International Conference on Learning Representations (ICLR 2022), 2022. URL iid= heet/forrum31d=_xxxrrpppss
Lowik Chanussot, Abhishek Das, Siddharth Goyal, Thibaut Lavril, Muhammed Shuaibi, Morgane Riviere, Kevin Tran, Javier Heras-Domingo, Caleb Ho, Weihua Hu, Aini Palizhati, Anuroop Sriram, Brandon Wood, Junwoong Yoon, Devi Parikh, C. Lawrence Zitnick, and Zachary Ulissi. Open Catalyst 2020 (OC20) Dataset and Community Challenges. ACS Catalysis, 11(10))6059-600 6072, 2021. doi: 10. 102 //cccatal 0c04525
Chi Chen and Shyue Ping Ong. A universal graph deep learning interatomic potential for the periodic table. Nature Computational Science, 2(11):718 1288 2022. doi: 00003///43388002220033-33333
Chi Chen, Weike Ye, Yunxing Zuo, Chen Zheng, and Shyue Ping Ong. Graph Networks as Universal Machine Learning Framework for Molecules and Crystals. Chemistry of Materials, 31 (9):3564-355) 3572, 2019, doi: 10.
Jiucheng Cheng, Chunkai Zhang, and Lifeng Dong. A geometric-informaioonn intormaiionnnnnaneneeeee crystal graph network for predicting properties of materials. Communicutions Materials, 2(1):92, 2021. doi: 10.10038/43344--2211 -0019433
Naoya Chiba, Yuta Suzuki, Tatsunori Taniai, Ryo Igarashi, Yoshitaka Ushiku, Kotaro Saito, and Kanta Ono. Neural structure fields with application to crystal structure autoencoders. Communications Materials, 4(1):106, 2023. doi: 10.00388444446222333332

---

Published as a conference paper at ICLR 2025
Kamal Choudhary and Brian DeCost. Atomistic Line Graph Neural Network for improved materials property predictions. npj Computational Materials, 7(1):185, 2021. doi: 10.1038/s41524-021 00650-1
Kamal Choudhary, Kevin F. Garrity, Andrew C. E. Reid, Brian DeCost, Adam J. Biacchi, Angela R. Hight Walker, Zachary Trautt, Jason Hattrick-Simpers, A. Gilad Kusne, Andrea Centrone, Albert Davydov, Jie Jiang, Ruth Pachter, Gowoon Cheon, Evan Reed, Ankit Agrawal, Xiaofeng Qian, Vinit Sharma, Houlong Zhuang, Sergei V. Kalinin, Bobby G. Sumpter, Ghanshyam Pilania, Pinar Acar, Subhasish Mandal, Kristjan Haule, David Vanderbilt, Karin Rabe, and Francesca Tavazza. The joint automated repository for various integrated simulations (JARVIS) for data-driven materials design. npj Computational Materials, 6(1):173, 2020. doi:
Weitao Du, He Zhang, Yuanqi Du, Qi Meng, Wei Chen, Nanning Zheng, Bin Shao, and Tie- Yan Liu. SE(3) Equivariant Graph Neural Networks with Complete Local Frames. In The 39th International Conference on Machine Learning (ICM1.2022), volume 162 of Proceedings of Machine Learning Research, pp. 5583-5608, 2022. URL httpps://ppp.ceeeeeetssss ngs. mlr
pzess/v162/du222 html
Weitao Du Yuanqi Du, Limei Wang, Dieqiao Feng, Guifeng Wang. Shuiwang Ji, Carla P Gomes, and Zhi-Ming Ma. A new perspective on building efficient and expressive 3D equivariant graph neural networks. In Advances in Neural Information Processing Systems (NeurIPS 2023), volume 36, pp. 66647-66674, 2023. URL wttp::ttt////t wwtp:p/rrrrrpp...ccceeeeeppeeeee htsppp/p...ppppppppeeeeeeeeeeeee psppppe:/ppppp html
Alexandre Duval, Simon V. Mathis, Chaitanya K. Joshi, Victor Schmidt, Santiago Miret, Fragkiskos D. Malliaros, Taco Cohen, Pietro Liò, Yoshua Bengio, and Michael Bronstein. A Hitchhiker's Guide to Geometric GNNs for 3D Atomic Systems, 2024. arXiv: 2312.07511
Alexandre Agm Duval, Victor Schmidt, Alex Hernández-García. Santiago Miret, Fragkiskos D. Malliaros, Yoshua Bengio, and David Rolnick. FAENet: Frame Averaging Equivariant GNN for Materials Modeling. In The 40th International Conference on Machine Learning (ICML 2023), volume 202 of Proceedings of Machine Learning Research, pp. 9013-9033, 2023. URL ml www.:mm...ppprrpeemmm / /202///22////// duva 1 123a .hmll]
Nadav Dym, Hannah Lawrence, and Jonathan W. Siegel. Equivariant Frames and the Impossibility of Continuous Canonicalization. In The st International Conference on Machine Learning (ICML 2024), volume 235 of Proceedings of Machine Learning Research, pp. 12228-12267, 2024. URL
mw..press v235/ dym24a. html
Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling. SEa3)-Transformer::
3D Roto-Translation Equivariant Attention Networks.
In Advances in Neural
Information Processing Systems (NeurIPS 2020), volume 33, pp. 1970-1981, 2020. URL https::/papeetteee nips cc wwtpccppppppppeeeeeteetcceeeeeee
files http://ppp//2///hha/// 15231a a wec.eba77790 3ttaa//8811112222cc33333333 oat.raae....cccceeaeaaaaaaaaaaaa Abstract.
Jan F. Gerken, Jimmy Aronsson, Oscar Carlsson, Hampus Linander, Fredrik Ohlsson, Christoffer Petersson, and Daniel Persson. Geometric deep learning and equivariant neural networks. Artificial Intelligence Review, 56(12): 14601-16661166 2023. doi: 10. 10.70 1442077/00462-223111000000 10502-7
Jaqqi Han, Jiacheng Cen, Liming Wu, Zongzhao Li, Xiangzhe Kong, Rui Jiao, Ziyang Yu, Tingyang Xu, Fandi Wu, Zihe Wang, Hongteng Xu, Zhewei Wei, Yang Liu, Yu Rong, and Wenbing Huang. A Survey of Geometric Graph Neural Networks: Data Structures, Models and Applications, 2024.
arXiv: 2200224433004
Xiao Shi Huang, Felipe Perez, Jimmy Ba, and Maksims Volkovs. Improving Transformer Optimiza- tion Through Better Initialization. In The 37th International Conference on Machine Learning (ICMI. 2020), volume 119 of Proceedings of Machine Learning Research, pp. 4475 4483, 2020. URL
ngs ml www..p..ss.vv....... patress/v119/hhagggeeee h//haagg200fff ..ttangg2f.f.
Pavel Izmailoy, Dmitrii Podoprikhin, Timur Garipov, Dmitry P. Vetrov, and Andrew Gordon Wilson. Averaging Weights Leads Wider Optima and Better Generalization. In The 34th Conference on Uncertainry in Artificial Intelligence (UAI 2018), pp. 876-885, 2018. URL http: bbtt:://aue /auai
12

---

Published as a conference paper at ICLR 2025
Anubhav Jain, Shyue Ping Ong, Geoffroy Hautier, Wei Chen, William Davidson Richards, Stephen Dacek, Shreyas Cholia, Dan Gunter, David Skinner, Gerbrand Ceder, and Kristin A. Persson. The Materials Project: A materials genome approach to accelerating materials innovation. APL Materials, 11))01 1002, 2013. doi: T0. 4814311188122233
Rui Jiao, Wenbing Huang, Peijia Lin, Jiaqi Han, Pin Chen, Yutong Lu, and Yang Liu. Crystal Structure Prediction by Joint Equivariant Diffusion. In Advances in Neural Information Processing Systems (NeurIPS 2023), volume 36, pp. 17464-17497.
2023. URL httpp:://proceeeeteete
neur http:///nwnnn.n...oo.....pppeepa
Aasrreoo-----ssstrtt-a-C-oooeeee Conffrect- htm.
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In The Third International Conference on Learning Representations (ICLR 2015), 2015. arXiv: 1412.6980
Scott Kirklin, James E. Saal, Bryce Meredig, Alex Thompson, Jeff W. Doak, Muratahan Aykol, Stephan Rühl, and Chris Wolverton. The Open Quantum Materials Database (OQMD): assess- ing the accuracy of DFT formation energies. upj Computational Materials, 1(1):15010, 2015. doi: 100.130//mpcopmmmss2055...
Yi-Lun Liao and Tess Smidt. Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphs. In The Eleventh International Conference Learning Representations (ICLR 2023), 2023. URLpttss:: / openreview. wwt. fort:/foum??e d= FARGOTD,
Yi-Lun Liao, Brandon M Wood, Abhishek Das, and Tess Smidt. EquiformerV2: Improved Equiv- ariant Transformer for Scaling to Higher-Degree Representations, In The Twelfth International Conference on Learning Representations (ICLR 2024), 2024. URL http:tt://oppereiiieee net Diet/forum20 2dd-mummmmmmmmm0mmmmm
Tianyang Lin, Yuxin Wang, Xiangyang Liu, and Xipeng Qiu. A survey of transformers. Al Open, 111-132, 2022. doi: 10. T0..0166/jaiopenn2222 2022. 10.001
Yuchao Lin, Keqiang Yan, Youzhi Luo, Yi Liu, Xiaoning Qian, and Shuiwang Ji. Efficient Ap- proximations of Complete Interatomic Potentials for Crystal Property Prediction. In The 40th International Conference on Machine Learning (ICML 2023), volume 202 of Proceedings of mlr
Machine Learning Research, pp. 21260-21287, 2023. URL
press/v202
lin23m..mmmmmmmm 200Ss//222//1n2mmm html
Yuchao Lin, Jacob Helwig, Shurui Gui, and Shuiwang Ji. Equivariance via Minimal Frame Averaging for More Symmetries and Efficiency. In The 4/ International Conference on Machine Learning (ICML 2024), volume 235 of Proceedings of Machine Learning Research, pp. 30042-30079, 2024. URL https: http::///occeeeieeeee mlr wwtppptssss......pppppp 1etp/iin44.... html
Chuang Liu, Zelin Yao, Yibing Zhan, Xueqi Ma, Shirui Pan, and Wenbin Hu. Gradformer: Graph Transformer with Exponential Decay. In The 33rd International Joint Conference Artificial Intelligence (IJCA/ 2024), pp. 2171-2179, 2024. Main Track. doi: 10.24963/ijcai.22244444
Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. In The Seventh International Conference on Learning Representations (ICLR 2019), 2019. URL https:/7
Steph-Y Yves Louis, Yong Zhao, Alireza Nasiri, Xiran Wang, Yuqi Song, Fei Liu, and Jianjun Hu, Graph convolutional neural networks with global attention for improved materials property prediction. Physical Chemistry Chemical Physics, 02:18141-18144 2020. doi: 10.1039/DDCCP11774
Paul Niggli. Handbuch der Experimentalphyssk. akademische Verlagsgesellschaft,, 1928.
Cheol Woo Park and Chris Wolverton. Developing an improved crystal graph convolutional neural network framework for accelerated materials discovery. Physical Review Materials, 4:063801,
2020. doi:
Sergey Pozdnyakov and Michele Ceriotti. tion for deep learning on point clouds.
Smooth, exact rotational symmetriza- In Advances in Neural Information Pro- cessing Systems (NeurIPS 2023), volume 36. pp. 79469-79501, 2023. URL
httpss//ppooceedngg...eni....... www.:ngs...uuurrooe Ficee3322253022aacccccceeeeaaaar
13

---

111108822662-2153/aal. Published as a conference paper at ICLR 2025
Sergey N Pozdnyakov and Michele Ceriotti. Incompleteness of graph neural networks for points clouds in three dimensions. Machine Learning: Science and Technology, 3(4):045020, 2022. doi: 10. 10088/263222153/aaaa
Omri Puny, Matan Atzmon, Edward J. Smith, Ishan Misra, Aditya Grover, Heli Ben-Hamu, and Yaron Lipman. Frame Averaging for Invariant and Equivariant Network Design. In The Tenth International Conference on Learning Representations (ICLR 2022), 2022. URL https://
openreview net/ Forum wwt.oummdd-iii0
Charles Ruizhongtai Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas. PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation. In 2017 IEEE Con- ference on Computer Vision and Pattern Recognition (CVPR 2017), pp. 77-85, 2017. doi: 20.T009/VVR.2217..16 URL htttps://ppeacccss evpr 2017/htm1/00 PointNet Deep Learning CVPR 2017 paper htmm
thecvf. ccmm content
A. Santoro and A. D. Mighell. Determination of reduced cells. Acta Crystallographica Section A, 26 1177124127,, 1970. doi: 10. 1110///056777777000071777
K. T. Schütt, H. E. Sauceda, P.-J. Kindermans, A. Tkatchenko, and K.-R. Müller. SchNet A deep learning architecture for molecules and materials. The Journal of Chemical Physics, 148(24): 241722, 2018. doi: 0..10063/.5019777
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-Attention with Relative Position Representa- tions. In The 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2018), volume 2 (Short Papers), pp. 464 468, 2018. doi: 10.1 10.18653/v///88-11
Yu Shi, Shuxin Zheng, Guolin Ke, Yifei Shen, Jiacheng You, Jiyan He, Shengjie Luo, Chang Liu, Di He, and Tie- Yan Liu. Benchmarking Graphormer on Large-Scale Molecular Modeling Datasets, 2023. arXiv: 04800
Yuta Suzuki, Tatsunori Taniai, Kotaro Saito, Yoshitaka Ushiku, and Kanta Ono. Self-supervised learning of materials concepts from crystal structures via deep neural networks. Machine Learning: Science and Technology, 344):045034, 2022.
Yuta Suzuki, Tatsunori Taniai, Ryo Igarashi, Kotaro Saito, Naoya Chiba, Yoshitaka Ushiku, and Kanta Ono. Contrastive Language-Struun Structure Pre-training Driven by Materials Science Literature. 2025. arviv: Siiv:2501.11219
Tatsunori Taniai, Ryo Igarashi, Yuta Suzuki, Naoya Chiba, Kotaro Saito, Yoshitaka Ushiku, and Kanta Ono. Crystalformer: Infinitely Connected Attention for Periodic Structure Encoding. In The Twelfth International Conference on Learning Representations (ICLR 2024), 2024. URL
htt.ps://ppenreviie.e
https: http:///ppwweeeee..eeee. www.riieeneeee.... finet/forum?ide fxiiii 19H
Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang. Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation- and translation-equivaat neural networks for 3D point clouds, 2018. arv: 1802.08219
Richard Tran, Janice Lan, Muhammed Shuaibi, Brandon M. Wood, Siddharth Goyal, Abhishek Das, Javier Heras-Domingo. Adeesh Kolluru, Ammar Rizvi, Nima Shoghi, Anuroop Sriram, Félix Therrien, Jehad Abed, Oleksandr Voznyy, Edward H. Sargent, Zachary Ulissi, and C. Lawrence Zitnick. The Open Catalyst 2022 (OC22) Dataset and Challenges for Oxide Electrocatalysts. ACS Catalysis, 3(5):3066-3084 2023. doi: 10. 102 1o.102//acccatal22055444
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz. Kaiser, and Illia Polosukhin. Attention is All you Need. In Ad- vances in Neural Information Processing Systems (NIPS 20/7), volume 30, pp. 6000-6010, 2017. URL httpps://ppooceddingg... neurips wwwpurrisscccpppeeeee coppaper files/ #iless/pape:2217// hash/3f5eee 3fsee24354444 www.:/557769991e1 /ww htt.://ee22244eeeeeeee000aaaaaa.
Abstract.
Yusong Wang, Shaoning Li, Tong Wang. Bin Shao, Nanning Zheng, and Tie-Yan Liu. Geometric Transformer with Interatomic Positional Encoding. In Advances in Neu- ral Information Processing Systems (NeurIPS 2023), volume 36, pp. 55981-55994,
14

---

Published as a conference paper at ICLR 2025
2023.
URL
hash/ ott.aaaaeaaaaaaaaaaaaaaaaaaaaaen
html
Tian Xie and Jeffrey C. Grossman. Crystal Graph Convolutional Neural Networks for an Accurate and Interpretable Prediction of Material Properties. Physical Review Letters, 120:145301, 2018.
doi: 10.1 10t//hysseee...ec.. 120. 145301
Keqiang Yan, Yi Liu, Yuchao Lin, and Shuiwang Ji. Periodic Graph Transform- ers for Crystal Material Property Prediction. In Advances in Neural Information Processing Systems (NeurIPS 2022), volume 35, pp. 15066-15080, 2022. URL
Keqiang Yan, Cong Fu, Xiaofeng Qian, Xiaoning Qian, and Shuiwang Ji. Complete and Efficient Graph Transformers for Crystal Material Property Prediction. In The Twelfih International Conference on Learning Representations (ICLR 2024), 2024. URL /tttps://openrvill
net/Eorum? 3003101S
Chengxuan Ying, Tianle Cai, Shengjie Luo. Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tin-Yan Liu. Do Transformers Really Perform Badly for Graph Representation? In Advances in Neural Information Processing Systems (NeurIPS 202 volume 34, pp. 28877-
28888, 2021. URL httpps://proceeddngg/ new.rreeerrpp.. cc wwtpaper_pppeee/ppppee Files/paper/
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhut- dinov, and Alexander Smola. Deep Sets. Advances in Neural Informa- tion Processing Systems (NIPS 2017), volume 30, pp. 3394-3404, 2017. URL
http:://prooeednngg.eee..... wtwpsurrrssccccppppeeo ccppppr httpp//pppppeeeeepppeeeeeeeeeeee 222677770aaa22755555554444411022
A LIMITATIONS OF UNIT-CELL--BASEE CRYSTAL REPRESENTATIONS
The conventional PCA frames explained in Sec. 2.3 implicitly assume a unique lattice representation such as the Niggli reduced cell (Niggli) 1928). Similarly, the lattice frames assume primitive cell and convert it to cell similar to the reduced one. Otherwise, these frames are affected by the arbitrariness unit cell representations, such as supercells and conventional cells.
Traditionally, primitive cells and conventional cells are used represent periodic structures. Primitive cells are defined as the smallest repeating units a lattice, having the minimum volume and containing only single lattice point within each cell. By following mathematical procedure primitive cells, their unique representations called reduced unit cells can be obtained (Santoro & Mighell 1970).
On the other hand, conventional cells defined unit cells that not necessarily primitive but are designed to exhibit symmetry in an easily understandable way. The notion of conventional cells is often illustrated by the face-centered cubic lattice and the body-centered cubic lattice.
Reduced cell
Conventional cell
Figure Al: Conventional cell (green) and Niggli reduced cell (blue) for a face-centered cube.
15

---

Published as a conference paper at ICLR 2025
Figure A1 compares conventional cell and the Niggli reduced cell of face- cubic structure. Examining the conventional unit cell easily reveals that it represents a cubic lattice, with atoms located at each corner and at each face center. However, this fact obscured in the reduced cell. Therefore, reduced cells can be said to sacrifice the interpretability of physically important information, such as symmetry, in order to uniquely represent periodic structures.
B COMPARISON TO EXISTING LOCAL FRAMES FOR MOLECULES
In the molecular modeling literature, several local frames have been proposed (Du et al. 2022 2023 Pozdnyakov & Ceriotti 2023). The concept of our dynamic frames, being both dynamic and local, is distinct from these frames for molecules, which are local but static. Below we discuss this perspective in more detail.
We first clarify the terminology regarding 'dynamic' and "static' in this context. We use 'dynamic' describe behavior that is influenced by the model's internal states estimated for given structure. For instance, interatomic interactions modeled within a GNN reflect these internal states and evolve dynamically layer by layer. Dynamic frames are designed align with these interatomic interactions. While the molecular modeling literature often uses 'dynamic' to describe temporally evolving structures, our work does not assume such temporal dynamics. Similarly, we use "static" describe behavior that is unaffected by the model's internal states.
Du et al. 2022) propose static edge-wise frames. These edge-wise frames, denoted as Fij [ei, e2, e3 using our notation, are 3 X orthogonal matrices defined individually for each edge (i.j). From Eq. 2 in their paper, the axes of Fij are defined as = unit(p: -P,), e2 = unit(p; x P,), and C3 C₁ C2, where unit(x:) = z 2/||=||| is L2 normalization. Here, the centroid of the structure is pre-shifted to the origin, as p p P using p N Σ, Pt. Thus, these frames are translation invariant, even though e2 appears to depend on absolute positions. However, performing such a global centroid shift for crystals is not straightforward due to their infinite periodicity, unless specific unit cell description is utilized.
Du et al. (0233) propose frame-based equivariant message passing using static edge-wise and node-wise wise frames. These edge-wise frames Fij are identical to those used in their earlier work (Du al. 2022) (see above). Their node-wise frames F; are defined similarly to Foj. but with P, replaced by the cluster centroid around i: p. [aC EjeN() Ps. Thus, the axes of F; are provided as
e, = unit(p: P.). C2 unit(p; x p.). and C3 = e, x E2. (See Eqs. 13 and 14 in their paper for the definitions.) To ensure translation invariance, these node-wise frames also rely on global centroid normalization. Moreover, when applied to crystal structures. their highly symmetric nature will often cause P. Ps. resulting in unstable frame construction.
Pozdnyakov Ceriotti 2023) propose ensemble of many 3-body interactions called the equiv- ariant noordinate-systen ensemble. For each target atom they construct many triplets of atoms (i.j.j") using pairs of neighbors (j.j') and then construct a local frame for each triplet as Fijs-. Although these triplet-wise frames are local, they do not reflect dynamic internal states of the model. Also, modeling 3-body interactions is computationally expensive.
Overall, these methods all employ specific types of static local frames, such node-wise (Du et al. 2023), edge-wise (Du et al. 2022 2023), or triplet-wise (Pozdnyakov & Ceriotti 2023) frames. None of them leverage the model's 's internal states for frame construction.
we further compare the proposed method using dynamic frames with its static counterpart variant, which is based on static local frames. The results in Tables and demonstrate the superior performance of the proposed dynamic frames, highlighting the conceptual difference between these two families of frames.
In Sec. 5.1
C DATASET SPECIFICATIONS
We use the following three sources of materials data for evaluations. They are all publicly available through a Python package (jarvis-tools) created by Choudhary al. 2020).
16

---

Published as a conference paper at ICLR 2025
The JARVIS-DFT 3D 2021 is collection of 55,723 materials provided by Choudhary et al. (2020)
and is accessible as dft 2aff_____________________ 20_2011 via jarvis-tools (or as dft_ 3d in older versions). These materi- als are annotated with various simulated properties using two DFT calculation methods, OptB
(OPT) and TBmBJ (MBJ). Following recent studies (Yan et al. 2022 2024 Lin et al., 2023 Taniai al. 2024), we use formation energy (format on__iennnrrpeeeeeeeeeeeeeeeeeeee peeaeetttte t om), total energy (ertb88ddwttom totalllllee tettttllaeeeeettleeee _energy,, bandgap
Bedd_wawdwdeeeeeeeeeeee and mb abjj bandgan), and energy above hull E hull (ehull) as regression targets.
The Materials Project (MP) database (Jain et al. 2013) is an online public materials database providing various synthetic materials and their DFT-calculatdd properties. We specifically use its snapshot collected by Chen et al. (2019), which contains 69,239 materials and is accessible as megnet via jarvis-tools. Following recent studies (Yan et al. 2022 2024; Lin et al. 2023 Taniai et al., 2024), we use formation energy (e_form), bandgap (gap pbe), bulk modulus (bu k modulus), and shear modulus (shear modulus) as regression targets. For bulk and shear modulus, we use the data splits provided by Yan al. (2022).
The Open Quantum Materials Database (OQMD) is another online public materials database by Kirklin et al. (2015). We specifically use its snapshot provided as ogmd_3d_no_ctt in jarvis-tools, which contains 817,636 materials with three DFT-calculated properties: for- mation energy (oqmái delta_ee e), bandgap Cogmd band band_gap). and energy above hull Coqmd_st sirdd_stae saanddstabiii 1 lityy. ). We use these properties as regression targets. We release our data splits along with our code.
D TRAINING SETTINGS
Table A1 summarizes the training settings for the JARVIS, MP, and OQMD datasets.
Specifically, for the JARVIS dataset, we optimize the mean absolute loss function using the Adam optimizer (Kingma& & Ba 2015) with (3,, 82) = (0.9, 0.98) and weight decay of 10-3 (Loshchilov & Hutter 2019). We employ the warm-up-free inverse square root scheduling (Huang et al., 2002)) for the learning rate, with the initial learning rate of 5.0 x 10 and decay factor of v 40000/ (4000 + t) according the total train steps t. The model weights normalization-fia free transformer architecture by Huang et al. (2020), which improves the training stability. The training is iterated for total of 2000 epochs with batch size of 256 materials. Stochastic weight averaging (SWA) (Izmailov et al. 2018) is adopted for model selection for testing and validation. Except for the increased number of epochs, we use the same settings with the baseline Crystalformer model (Taniai et al.) 2024) to evaluate the pure effects of introducing the frames.
initialized through the strategy for the
For the OQMD dataset, which was not used by the baseline method, we use similar settings with a larger batch size of 1024 materials and fewer epochs of 200.
Table Al: Detailed training settings.
Hyperparameters
Settings (JARVIS, MP, OQMD)
Loss function
Mean absolute error
Optimizer
AdamW with (31.3)) = 00.9.0.98
Weight decay
10-5
Gradient norm elipping
1.0
Initial learning rate a
5.0 x 10-1
Learning rate scheduling step
ay (0000((0000+ (1)00++)
Warm-up steps
(no warm-up)
Batch size
256. 128, 1024
Number of epochs
2000, 800, 200
Dropout rate
0.0
SWA epochs
50, 50, 20
E FULL BENCHMARK RESULTS
Tables A2 and A3 provide the full versions of Tables D and 2 adding the results of CGCNN (Xie &
Grossman 2018), SchNet (Schütt al.) 2018), MEGNet (Chen et al. 2019). GATGNN (Louis et al.
17

---

Published as a conference paper at ICLR 2025
2020), M3GNet (Chen & Ong 2022). and ALIGNN (Choudhary & DeCost 2021). These tables
also include the comparisons between the default and lightweight configurations CrystalFramer described in Appendix G
We also report additional results for the OQMD dataset using larger models in Table A4] number of self-attention blocks enhances the performances of both the baseline and proposed models. In both the four and eight block models, the version with max frames consistently outperforms the baseline. Investigating even larger models is left for future work.
Increasing the
F DETAILED VISUAL ANALYSIS OF FRAMES
F.1 COMPARISON BETWEEN WEIGHTED PCA FRAMES AND MAX FRAMES
As shown in Tables and the max frame method performed very well, while the weighted PCA variant did not. In Fig. the weighted PCA frames do not seem to capture the local structure very well compared to the max frames. This is because all the attention weights, even small ones, can influence the composition of the weighted PCA frames. In other words, the weighted PCA frames look at the structure over broader area, while the max frames focus on relatively close neighbors. This difference seems have positive effect on the frames and negative effect on the weighted PCA frames in most tasks, except for the E hull in the MP dataset (Table
For the E hull prediction, it is suggested by Taniai al. (2024) that the inclusion of long-range interatomic interactions is a critical factor. This implication can reasonably explain the better performance the weighted PCA frames for the hull. That is, the weighted PCA frames emphasize distant atoms and help deliver more meaningful messages from these distant atoms that are important for the E hull prediction.
F.2 FRAME VISUALIZATIONS FOR A DIFFERENT MATERIAL
Figure A2 shows the frame visualizations for another test material (IVASP-85272), This structure consists of carbon (red atoms) and nitrogen (blue atoms), forming a tetrahedral structure. Both dynamic models first attend to the central tetrahedral structure in the first two layers, and then increase the attention to relatively distant red atoms in the subsequent layers. However, the max frames capture these structures more clearly than the weighted PCA frames, observed in the first example.
We have also noticed general tendency for our models to attend to close neighbors in shallow layers and relatively distant neighbors in deeper layers. This tendency is also reasonable. Since the states of atoms are initialized as symbolic atomic species without rich information, they must gather information about their surroundings in shallow layers configure their states. In deeper layers, these atoms become ready to engage in complex interactions with selected distant atoms.
Crystal structure (C,N.)
Weighted PCA
PCA frame
Lattice frame
Layer
Layer
Layer
Layer
Figure A2: Frame visualizations for a different material.
18

---

Published as a conference paper at ICLR 2025
Table A2: Property prediction results on the JARVIS dataset (full).
Form. energy 54778/587/5572
Total energy
Bandgap (OPT)
Bamigap (MBJ) 14597/1817/1817
hull
44876/552//572
44578/5522/55 5572
44796/537//537
Method
CGCNN Xic Grossman
0.063
SchNet Schult al
2018
0.045
0.047
0,19
0.43
0,14
MEONET Den 2019
0.047
0,058
0,145
0,34
0.084
GATGNN
2020
0.047
0.056
0.17
0.51
0.12
M3GNNet
Ten & Ung
2022
0.039
0.041
0.145
0,362
0.095
ALIGNN
Matformer Yan claai
0.0325
0.035
0.137
0.30
0.064
af
2725
0.0294
0.032
0,127
cComFormer
Yan et
2024
0.0284
0.032
0.124
0.28
0.044
iComFoomer
Yan ct
2024
0.0272
0,0288
0.122
0.26
0.047
Crystalformer
Tanial ct
2024
0.0306
0.0320
0.128
0,274
0.0463
PCA frames
Divvl
2023
w/ lattice frames
Yau ct
2024
0.0302
0.0323
0,125
0.274
0.0531
static local Franes
0.0285
0,0292
0.122
0.261
0.0444
weighted PCA frames (proposed)
0.0287
0.0305
0,126
0.279
0.0444
w/ max frames (proposed)
0.0263
0.0279
0.117
0.242
0.0471
CrystalFramerr (default)
0.0263
0.0279 0.0279
0.117
0.242
0.0471
CrystalFramerr (lghtweight)
0.0268
0.117
0.262
0.0467
Table A3: Property prediction results on the MP dataset (full).
Fonnation energy 60000 5000/439
Bandgup 6000/ 5000/4239
Bulk modulus 4664/ 333/393
Shear muduluu
4664/ 392/
Method
eV/atom
ev
logrCiPa)
log(GiPa)
CGCNN
0.031
0,292
0.047
0.077
SchNet
0.033
0.345
0.066
0.099
MEGNet
0.030
0.307
0.060
0.099
GATGNN
0.033
0,280
0.045
0.075
M3GNet
0.024
0.247
0.050
0.087
AT IGNN
0.022
0.218
0.051
0.078
Matformer
0.021
0.211
0.043
0.073
PotNet
0.0188
0,204
0.040
0.065
Cormerree
0.0182
0,202
0.0417
0.0729
Former
0.0183
0.193
0.0380
0.0637
Crystalfommer
0.0186
0.0377
w/ PCA frames
0.0197
0.217
0.0424
0.0719
lattice frames
0.0194
0,212
0.0389
0.0720
w/ static local frames
0.0178
0.191
0.0354
0.0708
w/ weighted PCA frames (proposed)
0.0197
0,214
0.0423
0.0715
frames (proposed)
0.0172
0.185
0.0338
0.0677
CrystalFramer (default)
0.0172
0.185
0.0338
0.0677
CrystalFramer (lighweight)
0.0176
0.191
0.0341
0.0654
Table A4: Property prediction results on the OQMD dataset (full).
Blooks
Form, energy (eVlatom)
Bandgap (cV)
hull (eVfatom)
Method
654108
653388
654108/81763/8/76
Crystaformmrr (haseline)
0.02115
0.06028
0.06759
CrystalFramer (default)
4
0.01871
0.05805
0.06607
CrystalFramer flighweight)
0.01813
0.05773
0.06672
Crystalfomer (baseline)
0.02104
0.05986
0.06690
CrystalFramer (default)
0.01778
0.05785
0.06454
CrystalFramer (lightweight)
0.01731
0.05142
0.06403
Table A5: Efficiency comparison with the lightweight configuration. See Appendix G for details.
Config.
Timelepoch
Test/mater.
Params.
+ Paramss/hlock
Default
7
16.8 mss
952 K
231 K
Lightweight
K
19

---

Published as a conference paper at ICLR 2025
Weighted PCA Frame
Max Frame
Figure A3: Evolution of dynamic frames during training. We visualize the weighted PCA frames and max frames using model checkpoints taken every 200 epochs, starting from epoch 100 until 2000. Frames from earlier checkpoints are overlaid with higher transparency. Notably, the max frames stabilize more quickly than the weighted PCA frames.
F.3 EVOLUTION OF DYNAMIC FRAMES DURING TRAINING
We further examined how dynamic frames evolve throughout the training process, by visualizing frames using model checkpoints taken at 200-epoch intervals. Figure A3 compares the evolution of the weighted PCA frames and max frames for the same material Fig. We observed that the weighted PCA frames fluctuated throughout training, whereas the max frames stabilized quickly during the early stages, As frame fluctuations can introduce noise, the early stabilization of the max frames may explain their superior performance compared to the weighted PCA frames.
G EFFICIENT MODEL. CONFIGURATION
As discussed in Sec. 5.2 the key improving the runtime of method is to accelerate the infinite summation of edge features: Σn Way(n) Pyj()- These edge features 4/sf(m)) are the sum of one distance-based feature and three angle-based edge features represented via GBFs b:
Cain Wobais (ris(n)) Cangl
Canaa
Σ 1,2,
Wabbagt (o(k) (i(m)) )
(A1)
Here, new coefficient parameters X, Caisa: Gangl E R2 have been introduced into the original form in Eq.7 The default model configuration uses largely overlapping Gaussian bases (s = 4.0) for angular features. This means that bang changes very smoothly along and can be well-approximated with fewer Gaussian bases (i.e., lower D).
When reducing D Dí, one can obtain comparable Gaussian widths and value ranges by converting their scale to 8' = sD' /D D and coefficient to = cD/ D'. Using this rule, we obtained lightweight version of the angular feature term by changing {D, s, c} from (64, 4.0, 1.0) (16, 1.0, 4.0}. After hyperparameter tuning, we further increased À from 1.0 to 1.5.
Table A5 compares the efficiencies of CrystalFramer models with the default and lightweight config- urations, and Tables A2 compare their performances the JARVIS, MP, and OQMD datasets. These results show that the lightweight model reduces the train time by 42% while achieving compa- rable accuracies on the JARVIS and MP datasets and better accuracies on the OQMD dataset.
H SCALABILITY FOR LARGE STRUCTURES AND SUPERCELLS
Since the proposed CrystalFramer is based self-attention mechanism, its computational complex- ity is O(Nk), where N is the number of atoms in the unit cell and is the number of neighbors per unit-cell atom. In the infinitely connected attention of Crystalformer (Taniai al. 2024) defined in
20

---

Published as a conference paper at ICLR 2025
Eq. B neighbors (n) are adaptively determined for each atom i in each layer. The current implemen- tation computes neighbors by periodically repeating the unit cell within a fimite range. Consequently, k becomes multiple of N, resulting in overall computational complexity of O(N?).
In practice, the training of CrystalFramer has successfully scaled to relatively large structures in the MP dataset, which features average of 30 atoms per unit cell and maximum of 296 atoms. For inference, the method can handle even larger structures than during training, as requires significantly less memory and supports per-material (non-batched) processing.
Scalability for larger structures becomes crucial especially when processing supercells. Supercells are often utilized when structures deviate from perfect periodicity, such in the presence of impurities, defects, or surfaces. We consider the following two potential approaches to improve efficiency with large supercells.
Mixed atom embedding. Structures with impurities or defects are often represented using site occupancy, which indicates the probabilities of different elements occupying an atomic site. Instead of modeling such structures with supercells, we can efficiently represent the site occupancy by mixing atomic embedding vectors. In this case, each o; represents a probability distribution over elements rather than single element. The corresponding atomic state can then be initialized as linear blend (element). This approach can keep the structure size small without using supercell, thereby maintaining overall efficiency.
of atom embeddings: æ, <- Eelement ay
Distance-based neighbor search. When unit cells are large, the current cell-based neighbor identifi- cation method will produce redundant neighbors, forcing k N. By employing more compact set of neighbors through nearest neighbor search, the complexity is reduced from O(N3) to O(Nk), improving efficiency for larger structures.
Since structures with imperfect periodicity are common in realistic scenarios, developing scalable models for these structures is important direction for future research.
I ANALYSIS OF MODEL'S S CONTINUITY
Dym al. 2024) pointed out that frame-based models generally exhibit discontinuous characteristics, which are also inherent in our approach. To empirically assess the degree of this discontinuity in our trained models, analyze the variations in their outputs for given crystal structure under perturbations.
The results in Figure A4 show that the weighted PCA frame model exhibits significantly smoother transition compared to the max frame model. However, as shown in Tables [ and 2 the weighted PCA frame method has lower performance, indicating that higher continuity does not necessarily translate to better performance. The discontinuous behavior of max frames may have facilitated the early stabilization of frames during training, discussed Appendix F3 leading to the superior performance.
Meanwhile, the discontinuity of the max frame model becomes more significant with larger pertur- bations. This trend suggests that the model may have limited generalization to out-of-domaii data. The technique of weighted frames proposed by Dym et al. (2024) could be applied improve the continuity of our max frame models.
21

---

Published as a conference paper at ICLR 2025
Input structure
Weighted PCA frame
Max frame
0.64
Network
0.62
0.0
030
0.6
0.5
-ho
0.1
-0.3
Atomic position perturbation rate
Atomic position perturbation rate
Figure A4: Continuity under perturbations. We examine the transitions in the outputs of trained BeaInPb (JVASP-70556) from
models for a test material under perturbations. Specifically, the JARVIS formation energy prediction task and perturb one of the beryllium (Be) atoms along the direction of lattice vector. The weighted PCA frame model shows smoother transition compared to the max frame model.
22


Reviews:
[{"review_id": "DW1JUwTlhg", "reviewer": "Reviewer_AkN2", "summary": "The authors change the fixed, static frames for SE(3)-invariant modeling in crystal structures into dynamic frames that adapt to each atom's local environment through the network's attention mechanism, then integrate into a new architecture **CrystalFramer**. Experimental results on large-scale datasets (JARVIS, MP, OQMD) demonstrate that **CrystalFramer** with dynamic frames outperforms models using static frames in several material property prediction tasks.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The proposed method is straightforward and presented with commendable clarity.\n2. The shift from static to dynamic frames enables atom-specific adjustments, significantly enhancing SE(3) invariance handling in highly symmetric crystal structures. Furthermore, the extensive evaluations on multiple datasets (JARVIS, MP, OQMD) highlight the effectiveness of dynamic frames, with the \"max frames\" variation demonstrating particular superiority.", "weaknesses": "### Major\nHow do the \"max frame\" construction and its usage within the networks differ from those proposed in [1], [2], [3], and [4]? In particular, [1] and [4] use similar atomic local frames to address equivariance in molecular structures. A comparative discussion of these methods would be valuable to clarify whether the proposed approach is a direct adaptation from molecular modeling to materials or if there are novel insights beyond those methods.\n\n### Minor\nUsing frame-based techniques will hinder the model's continuity, potentially affecting its generalization capabilities, as noted in [5]. Does the proposed model address this limitation? It will be valuable to discuss the continuity of the proposed dynamic frame approach and empirically analyze if this affects generalization in the experiments.\n\n\n\n[1] \"SE (3) equivariant graph neural networks with complete local frames.\" Du, et al.\n\n[2] \"Smooth, exact rotational symmetrization for deep learning on point clouds.\" Pozdnyakov, Sergey, and Michele Ceriotti. \n\n[3] \"Equivariance via Minimal Frame Averaging for More Symmetries and Efficiency.\" Lin, et al. \n\n[4] \"A new perspective on building efficient and expressive 3D equivariant graph neural networks.\" Du, et al. \n\n[5] \"Equivariant frames and the impossibility of continuous canonicalization.\" Dym, Nadav, Hannah Lawrence, and Jonathan W. Siegel.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Wti30qlHd5", "reviewer": "Reviewer_bzKy", "summary": "This paper proposes CrystalFramer and dynamic frames to capture SE(3)-invariant geometric features for crystal property prediction. Unlike using static frame in previous work, this work use dynamics frames to give each atom its own dynamics view of the structure, focusing only on those atoms actively interacting with it, meanwhile also accommodating the infinite and symmetric nature of crystal structures. The experiment shows superior performance of the proposed method in several prediction tasks.", "questions": "- Please refer to the weakness part.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This work proposes dynamic frames for crystal features modeling, which can dynamically accounting for the atoms actively engaged in learned interactions in each interatomic message-passing layer.\n- The proposed method achieves superior performance on several commonly used benchmarks in crystal property prediction.", "weaknesses": "- Previous relevant frame-based method ComFormer also provides equivariant version. I am wondering if this method can also be easily extended to SE(3)-equivariant? There’re are many other equivariant properties for the materials such as force or high-order tensors. I would like to see the performance of equivariant version of this method if the extension is straightforward. Even for the invariant properties, it’s interesting to know whether equivariant network can help invariant properties prediction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZkWyqVrb8V", "reviewer": "Reviewer_SjyW", "summary": "The paper proposes a new method for modeling crystal based on dynamic frames that are based on the local neighborhood of each atom in the crystal structure. The crystal frames are then processed by a transformer based architecture called CrystalFramer. The paper starts by motivating the utility of geometric deep learning methods that can capture relevant physical inductive biases, such as invariance. Previous work captured such geometric information using frame averaging, which is based on taking canonical frames of the entire structure. By contrast, the paper proposes dynamic frames which are based on the local coordinate system of a given atom. Section 2 then outlines relevant preliminary details with relevant mathematical definitions, including the description of a crystal structure, how transformers have been applied to crystal structure modeling and prior work on frame averaging for molecular and materials modeling.  \n\nSection 3 introduces and defines the concept of dynamic frames, which is the main part of the proposed new method. This includes the frame definitions in Section 3.1 and the CrystalFramer architecture in Section 3.2 that is based on Gaussian basis functions. This is followed by a description of related work on invariant features, frames and equivariant features in Section 4. \n\nSection 5 outlines the main experiments in the paper, including materials property prediction and an efficiency analysis . The results generally show that the proposed dynamic frames improve Crystalformer modeling performance across all datasets while achieving best performance in most cases. The efficiency analysis shows that CrystalFramer incurs an extra compute cost. Section 6 provides a visual analysis of the dynamic frames, as well as discussion on limitations along with suggestions for using dynamic frames for equivariant prediction and molecular modeling.", "questions": "* Could you talk more about scalability of your method? This would be especially relevant when modeling larger supercell that have more atoms.\n* Did you notice an evolution of the dynamic frames as training progressed? Are there consistencies across some types of materials?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "* The idea of dynamic frames proposed in the paper is a novel and effective way to include relevant inductive biases for crystal structure modeling.\n* The paper is well written with relevant details described and convincing experiments followed by detailed analysis.\n* The paper candidly discusses advantages and limitations of the proposed method.", "weaknesses": "* The paper could be further with additional discussion of related work for machine learning potentials [1 egraff]. The current focus of the study is mainly on properly modeling, so this would probably fit as potential future work. There be advantages and disadvantages of using CrystalFramer in such a context. Similarly, the paper can also discuss potential application to modeling of crystals with surfaces, such as catalyst [2 - OCP].\n* The dynamic frames provide a new dimension for analysis for how local neighborhoods interact in geometric deep learning. The paper could be strengthened by discussing the frames across different materials and/or different stages of the training process.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rG4bQF4GVQ", "reviewer": "Reviewer_dkNi", "summary": "This work explores an alternative, more powerful version of frame averaging networks and applies it to crystal property prediction tasks. Instead of using a fixed lattice frame or one calculated from the PCA of the moment tensor of the atoms, the authors propose to learn \"dynamic frames\" for each individual atom. They propose two methods to do this: Either by calculating the PCA from weighted atoms or constructing the frames by greedily choosing the highest weighted atoms, where the weights in both cases are equal to the attention weights in their transformer architecture. They show that this results in significant performance gains and that the model exhibits intuitive, sensible behaviour, where the earlier layers attend to closer atoms, and later layers attend to further away atoms.", "questions": "In the runtime comparison, Table 4: How can PotNet take only 43s/epoch to train but 313ms for inference, while CrystalFramer takes almost double the time to train with 74s but is nearly ten times faster during inference? Can you explain this large discrepancy?\n\nIn line 298: The model uses the cosine of the angles. This would map pi/2 and -pi/2 to the same value 0. Have you considered using a concatenation of both sine and cosine of the angles as features? It might make the model more expressive.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Crystal property prediction is an important topic, and (stochastic) frame averaging is a promising direction for the task. Using the novel idea of dynamic frames, the proposed method improves stochastic frame averaging for crystal property prediction by a large margin. Therefore, the contribution seems significant and novel. It is believable that the proposed \"dynamic frames\" make the model more expressive, although I find it hard to get an intuitive understanding of what the angles between the learned frames and direction vectors could learn or why they could represent useful features. The experimental results show strong improvements on two datasets and against the base model (Crystalformer) and other state-of-the-art models.", "weaknesses": "The biggest weakness of this paper is, in my opinion, the writing. Many sentences are grammatically incorrect or use strangely constructed, overly long sentences. This makes it much harder to read the paper than it should be. A non-exhaustive list of examples:\n\nLine 121: \"... are linear projects of current state x, σi is a tail-length variable\nof Gaussian distance-decay attention adaptively derived from xi, ψij(n) is geometric position\nembedding that encodes interatomic\". This misses several articles, and I assume the authors mean \"projections\" not \"projects\"\n\nLine 198: \"... and local view of entire crystal ...\" misses the article \"the\"\n\nLine 234: \"This perturbation scheme is considered to implement the stochastic FA\" I am not sure what this sentence means\n\nI strongly recommend that the authors use a grammar tool like Grammarly or an LLM to help streamline their writing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)