A Periodic Bayesian Flow for Material Generation
Authors: Hanlin Wu
Yuxuan Song
Jingjing Gong
Ziyao Cao
Yawen Ouyang
Jianbing Zhang
Hao Zhou
Wei-Ying Ma
Jingjing Liu
Venue: ICLR 2025 Spotlight (2025)
Keywords: Crystal Generation
Bayesian Flow Networks
Crystal Structure Prediction
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: We propose a periodic Bayesian flow with a novel mechanism to generate material under entropy guidance, achieving consistent better performance and significant higher sampling efficiency.

Abstract: Generative modeling of crystal data distribution is an important yet challenging task due to the unique periodic physical symmetry of crystals. Diffusion-based methods have shown early promise in modeling crystal distribution. More recently, Bayesian Flow Networks were introduced to aggregate noisy latent variables, resulting in a variance-reduced parameter space that has been shown to be advantageous for modeling Euclidean data distributions with structural constraints (Song, et al.,2023). Inspired by this, we seek to unlock its potential for modeling variables located in non-Euclidean manifolds e.g. those within crystal structures, by overcoming challenging theoretical issues. We introduce CrysBFN, a novel crystal generation method by proposing a periodic Bayesian flow, which essentially differs from the original Gaussian-based BFN by exhibiting non-monotonic entropy dynamics. To successfully realize the concept of periodic Bayesian flow, CrysBFN integrates a new entropy conditioning mechanism and empirically demonstrates its significance compared to time-conditioning. Extensive experiments over both crystal ab initio generation and crystal structure prediction tasks demonstrate the superiority of CrysBFN, which consistently achieves new state-of-the-art on all benchmarks. Surprisingly, we found that CrysBFN enjoys a significant improvement in sampling efficiency, e.g., 200x speedup (10 v.s. 2000 steps network forwards) compared with previous Diffusion-based methods on MP-20 dataset.

Published as a conference paper at ICLR 2025
A PERIODIC BAYESIAN FLOW FOR MATERIAL GENER- ATION
Hanlin Wul Yuxuan Song 1.3+ Jingjing Gong'
Ziyao Cao 1.
Yawen Ouyang'
Jianbing Zhang' Hao Zhou Wei-Ying Ma
Jingjing Liu
Institute of AI Industry Research (AIR), Tsinghua University
2 School of Vehicle and Mobility, Tsinghua University
3 Dept. of Comp. Sci. & Tech., Tsinghua University
4 School of Artifcial Intelligence, Nanjing University
wath 1 24 @m i 1 s. c ngghua.. edu. cr,
Is tsogyuxuan, zhouhao, ouyangyawen,
edu, cn
ABSTRACT
Generative modeling of crystal data distribution an important yet challenging task due the unique periodic physical symmetry of crystals. Diffusion-basee methods have shown early promise in modeling crystal distribution. More recently, Bayesian Flow Networks a variance-reduced parameter space that has been shown to be advantageous for modeling Euclidean data distributions with structural constraints (Song et al. (0223) Inspired by this, we seek to unlock its potential for modeling variables located in non-Eucline oon-Euclidenn manifolds e.g. those within crystal structures, by overcoming challenging theoretical issues. We introduce CrysBFN, novel crystal generation method by proposing a periodic Bayesian flow, which essentially differs from the original Gaussian-based BFN by exhibiting non-monotonic entropy dynamics. To successfully realize the concept of periodic Bayesian flow, CrysBFN integrates a entropy conditioning mechanism and empirically demonstrates its significance compared to time- come-conditioning. Extensive experiments over both crystal ab initio generation and crystal structure prediction tasks demonstrate the superiority of CrysBFN, which consistently achieves new state-of-the on all benchmarks. Surprisingly, we found that CrysBFN enjoys significant improvement in sampling efficiency, e.g., 200x speedup (10 v.s. 2000 steps network forwards) compared with previous diffusion-based methods on MP-20 dataset. Code available /ouon/wwwua wu han wwwhhaannnnnnnnnieeeeeeeeeeieeee WinnCryySBFN
introduced aggregate noisy latent variables, resulting in
attps: thub.. com
INTRODUCTION
Deep generative models, with their strong ability to approximate data distribution with complex geometries, have recently emerged as promising approach to de novo drug design (Hoogeboom al. 2022), protein engineering (Shi et al. 2022), and material science (Liu al. 2017). To discover
new functional materials (Wang et al.) 2023 : Peng et al., 2022). there has been an active line of
research on crystal generative modeling (Ren et al. 2021
Hoffmann et al. 2019 Noh et al. 2019
Court et al. 2020: Yang et al. 2023 Nouira al.
2018 Recent diffusion-baser models learns
through an iterative reverse process with multi-level noise perturbation, and has been demonstrated powerful tool for capturing complex geometries of crystals. Studies show that these models can
generate crystal samples with realistic structures that well satisfy physical constraint (Xie et al. 2021
Jiao al. 2023 2024
Lin et al. 2024).
Despite promising results, significant challenges persist. The search space for crystal structures grows exponentially with the number of atoms, while thermodynamically stable materials represent only a small fraction (Miller et al., 2024). This presents challenges in the multi-step generation process, the variance of which might cause structures deviate from stable distributions. Moreover,
"Equal Contribution. Correspondence to Hao Zhou (zhouhao air, www..iiiinnnaaidce..oooooo cdw..uuaa..uuummmmmm... and Yawen Ouyang (ouyangyawen www.swwwwwwetiiiining.....ooom

---

Published as a conference paper at ICLR 2025
Training Sampling
Belief Update
Periodic Equiv. Bayesian Flow
Pr(MM)
Perfodic Equalv. Network Defi MP()
(o(
A categurical
DxL
do
Blyymian Updare T()
0₂
M
vou (this paper)
Pertodic Equit. Buvesian Flow
MMP((M(
L Gaussian
Figure 1: Framework of CrysBFN. Left: overview of training and sampling process. At training time, the network receives 0;-1 from Bayesian flow based on data distribution, and tries to improve the belief 0 over the groundtruth MM by outputting an estimated distribution po and minimizing the gap between estimation and groundtruth. At sampling time with the trained network, the uninformative prior On is gradually improved by belief updates until O, with high fidelity. Right: illustration the periodic equivariant Bayesian flow.
the current widely-adapted diffusion- approaches (Jiao et al. 2023 2024) for crystal structure
modeling tend learn the score function of wrapped normal distribution for periodic variables, where the approximation of a sum of infinite terms is needed which could bring in extra bias. Recently, BFN (Graves et al., (2023) has been successfully applied the geometry generative modeling of molecules (Song et al. 2023), scenario that shares the above-mention meneeneet challenges similar to crystal much lower variance parameter space. However, the periodic geometry of crystals differs from that of small molecules and raises significant challenges.
generation, by modeling
To tackle these challenges, this paper aim to break the barrier of extending the paradigm of BFN into those variables located non-Euclidean space, e.g., atom fractional coordinates in crystal structure. We introduce the first non-Euclidean Bayesian flow over the periodic space, i.e. the hyper-torus. To successfully implement such concept, introduce generalized training paradigm based on simulation of the Bayesian flow and further propose a non-auto-reereessvv equivalent formulation of Bayesian flow distribution that guarantees computational efficiency. By integrating all these innovations, we introduce CrysBEN, the first periodic E(3) equivariant Bayesian flow network designed for crystal generation. Extensive experiments demonstrate the significant superiority of CrysBFN over current methods in both sampling quality and efficiency.
Our contributions can be summarized as follows:
We present the first periodic Bayesian flow in non-Euclidean space (hyper-torus) with novel training paradigm and entropy conditioning mechanism tackling the unprecedented and pivotal non-additive accuracy theoretical challenge.
We introduce the first periodic-E(3. equivariant Bayesian flow networks for crystal genera- tion tasks with appealing theoretical guarantees.
Extensive experiments demonstrate that CrysBEN consistently outperforms previous meth- ods on both ab initio crystal generation (99.1 1% COV-P Carbon-24) and crystal structure prediction tasks (64.35% match rate on MP-20). Efficiency experiments on MP-20 prove that CrysBFN enjoys a 200x sampling efficiency with performance on par with previous Diffusion- based methods.
RELATED WORK
Modeling and generating stable materials with data-driven approaches has been applied to discovering new functional materials (Peng et al. 2022 One line of approaches indirectly models crystal space by transforming crystals into human designed representations (Ren et al. 2021 Hoffmann et al. 2019; Noh et al. 2019: Court et al., 2020 Yang et al. (023),, though the encoding and decoding process often leads to physical geometry loss In contrast, another line of research directly models crystals in the sample space, drawing inspiration from the success of Diffusion models (Ho et al.)
2

---

Published as a conference paper at ICLR 2025
2020b; Song al. 2020al Song & Ermon 2019). For instance, CDVAE (Xie et al. 2021 and SyMat (Luo et al. 2024b) employ score- motechat (Song & Ermon 2019 to learn scores for generating stable materials, while their modeled distribution lacks geometric invariance (hhag al. 2023). DifCCSP (Jiao et al., 2023) addresses this by transforming Cartesian atom coordinates into fractional coordinates, introducing the periodic E(3) equivariance of crystals, and designing an equivariant diffusion crystal generation model based on periodic diffusion (Jing et al. (2022). More recently,
Miller al. (2024) applies Riemannian Flow Matching (Chen & Lipman (023) to crystal generation tasks offering improved sampling efficiency, while at the expense of quality y However, we argue that these methods struggle balance sampling quality and efficiency due to insufficient guidance during each transition from the noise prior to the data distribution. This issue is particularly pronounced for crystals, where thermodynamically stable materials constitute only a small fraction of the search space (Miller et al.) 2024). For instance, early generation states Ft-1 with low confidence should be retained less than later states to achieve the next state Zt--
In this work, we propose to BFN GGrves et al. 2023) model crystals in principally dif- ferent way. BFN provides a framework to accurately update each generation state according to its entropy/confidence, the effectivvness of which has been proved in Song et al. (023). However, there are no established explorations on the challenging topic of non- Euclidean BFN which is essential to many real-world applications Jing et al. (2022); Jumper al. (2021). To address the above issues, in this paper, we build a non- nun-Fuclidean Bayesian flow from scratch, identifying and tackling the non-additive accuracy via introducing a novel entropy conditioning mechanism.
3 PRELIMINARIES
Crystal Representation and Related Manifold Crystals can be represented as a structure composed of infinite, periodic and repeating unit cells defined by a triplet M = (A, F. L). Denote the number of atoms in the unit cell as N, A = (a1, a2...., aan E SKxN XN is the representation of atom types with the length of vocabulary K, and every such one-hot discrete variable locates in the simplex S. represented by:
SK ESER = 20 0, L..., K}
(I)
which requires the designed generative path for A should be well defined on the simplex. Following Jiao et al. 2003)). F = ff, , fN] E [0, 1)3xN is the fractional coordinates of atoms located in quotient space R 3x /23x equivalent to the hypertorus T (Jing al. 2022). The hypertorus
T3xN N can be represented as the Cartesian product of 3 x N toruses T
T def
{z RR:
|| = 1}
(2)
And L 12, E 3x3 denotes the lattice matrix, every column vector of which is the periodic basic vector of the crystal. We can get the Cartesian coordinates representation of unit cell's atom coordinates X by X LF E B The ideal infinite periodic crystal structure of M can be represented by {aa,, la = O; x = æ; + Lk, vk E Z3x1]. Based on the above notations, the symmetry of crystal geometry is defined as periodic E(3) invariance (Jiao et al. 2023). including periodic translational invariance of F and rotational invariance of L (details in Appendix B
Bayesian Flow Networks Different from the well-established SDE-based approaches, e.g. Diffusson Models (Ho et al.) 2020a Song et al., 20206 a) and ODE-based approaches e.g. Flow Match- ing (Lipman et al.) 2022). Bayesian Flow Networks define generative process driven by consecutive Bayesian updates on noised samples from the uninformative prior distribution Gu to posteriors 0
with higher confidence and more information.
To define the consecutive Bayesian update process, Bayesian Flow Networks contain a process to add noise to the clean data samples which is an analogy to the forward process Diffusion
For more detailed discussion of related work, please refer to Appendix F
2 "Actually some crystal structures are chiral (Flack 2003) and their structures are SE(3) invariant. In this paper, we follow Jiao et al. (2023) to build an E(3) invariant framework for fair comparison, while the SE(3) equivariance can be simply achieved by transforming by excluding invariance to reflections in the Markov chain.
F(3) invariant network into an SF(3) equivariant network
3

---

Published as a conference paper at ICLR 2025
Receiver Accuracy (log c)
5
Receiver Mean (m)
2
0
-5
-10
-15
0.0
0.2
0.4
0.6
0.8
1.0
0.2
0.6
0.8
1.0
Time (t)
Time (t)
Figure 2: Visualization of the proposed periodic Bayesian flow with mean parameter je and accumulated accuracy parameter which corresponds to the entropy//nceetaainy: For 0.3, 8(1) = 1000 and a; defined in Appendix A this figure plots three colored stochastic parameter trajectories for receiver mean parameter m and accumulated accuracy parameter c, su- perimposed on log-scale heatmap of the Bayesian flow distribution pr (m]2 a1, Q22,..., a;) and pr(ckz, 011, 02,..... a.). Note the non-monotyy and non-additive property of which could inform the network the entropy of the mean parameter m as a condition and the periodicity of m.
Models. In BFN, such process is explicitly defined by the so-called sender distribution ps (y|x; a) where a is the parameter of sender distribution which corresponds noise level, e.g. variance for Gaussian- ps-
BFN aims to create a procedure that gradually acquires information from the ground truth data x to pro- vide training signal. To this end, the framework will sample series of noisy samples Y2, Yn independently from the sender distribution ps with various accuracy levels a' 02 ...y Onn Based on samples, the framework would simulate outo-regressiv update of 0 based on the Bayesian update function to reflect an information gathering process from prior to data as:
0, 1,
The neural network V trained in teacher-forcing fashion: approximate the sender distribution ps(-|x; a;) generating y; according to 0;- which has integrated the information of (y1,... yi-1) through Eq. (3). The network hence takes 9 obtained by Bayesian update as input, and the distribu- tion implied by pr (-10) is termed input distribution. The network output (r(0) is interpreted as the parameter of an updated distribution Po( (-| (0)) over sample space, referred to as oulpui distribution. Combining the network output y (0) and the sender distribution, obtain the approximation to ps (y: x; a.) as:
(n 0,-1, VV, a.) = Eo (x11=(0: (y; x';
(4)
Such distribution is named as receiver distribution pn. Note that 0,-1 can be seen as deterministic function mapping, given update function in Eq. (3). By combining the objectives of different timesteps and taking expectation over trajectories, we obtain the training objective of BFN
0:- f(yl.... Yi-1) 1) based on the deterministic
L(V Ex--pan (x k (ys |x x, ax)llp((y: 0;-1, V a;))
(5)
Here Pdata is the empirical distribution. Detailed derivation and discussion are provided in Appendix Additionally, we provide toy examples with minimal components illustrating how BFNs work in our code repository.
4
METHOD
In this section, we explain the detailed design of CrysBEN tackling theoretical and practical challenges. First, we describe how to derive our new formulation of Bayesian Flow Networks over hyper-torus TD from scratch. Next, we illustrate the two key differences between CrysBFN and the original form of BFN: 1) meticulously designed novel base distribution with different Bayesian update rules; and 2) different properties over the accuracy scheduling resulted from the periodicity and the new Bayesian update rules. Then, we present in detail the overall framework of CrysBFN over each manifold of the crystal space (i.e. fractional coordinates, lattice vectors, atom types) respecting periodic E(3) invariance.
4

---

Published as a conference paper at ICLR 2025
AdditiveAoc. Bayesian Update
Non-addittveeAc. Bayeslan Update ontheTona
pla)
prev. receiver belief 0
prev. receiver belief 0/-1
observed sender sample y
+ observed sender sample y
= updated receiver belief 0
updated receiver belief 0
Figure 3: An intuitive illustration of non- non-additive accuracy Bayesian update the torus. The lengths of arrows represent the uncertainty/entropy of the belief (e.g. 1/o² for Gaussian and e for von Mises). The directions of the arrows represent the believed location (e.g. l for Gaussian and for von Mises).
4.1 PERIODIC BAYESIAN FLOW ON HYPER-TORUS TD
For generative modeling fractional coordinates in crystal, we first construct periodic Bayesian flow on T by designing every component of the totally new Bayesian update process which we demonstrate be distinct from the original Bayesian flow (please see Fig.
The fractional atom coordinate system (Jiao et al. 2023) inherently distributes over hyper- torer-torus support T Hence, the normal distribution support on used in the original (Graves al. 2023 is not suitable for this scenario.
To tackle this problem, the circular distribution (Mrddia & Jupp 2009) over the finite interval [-mr R) is a natural choice as the base distribution for deriving the BFN on TD, Specifically, circular distributions enjoy desirable periodic properties: 1) the integration over any interval length of equals 1; 2) the probability distribution function is periodic with period 2x. Sharing the same intrinsic with fractional coordinates, such periodic property circular distribution makes it suitable for the instantiation of BFN's input distribution, in parameterizing the belief towards ground truth on T
von Mises Distribution and its Bayesian Update We choose von Mises distribution (Mardia & Jupp 2009) from various circular distributions as the form of input distribution, based the appealing conjugacy property required in the derivation of the BFN framework. That is, the posterior of a von Mises distribution parameterized likelihood is still in the family of Mises distributions. The probability density function von Mises distribution with mean direction parameter m and concentration parameter (describing the entropy//ncertainty of m) is defined as:
[x c) = v.Maamm c) exp(c cos - m))
2x To(e)
where lo(c) is zeroth order modified Bessel function of the first kind the normalizing constant. Given the last univariate belief parameterized by von Mises distribution with parameter 0,-1 {m--1, } and the sample y from sender distribution with unknown data sample and known accuracy a describing the entropy/uncertainty of y, Bayesian update for the receiver is deducted as:
h((m,-1, 1 1), y, 0x) {m, ) where
m, = atan2 sin + 6;- sin m, a cos C-11 COs m4-1)
(8)
G == Vo 0 c²_ 20c; cos(y m:-1)
The atan2 function refers to 2-argument arctangent. Independently conducting Bayesian update for each dimension, we can obtain the Bayesian update distribution by marginalizing y:
The proof of the above equations be found in Appendix A.3
Pu (0'10, x; ar) E 50' - 1,0,,,a)) E Eesstyx. - ((0,y,a))
(10)
Non-additive Accuracy The additive accuracy is a nice property held with the coussian-formel sender distribution of the original BFN expressed as:
Pe (0" x; Ca + oz)) = Ep,, (0 1.x. (0" x1)
(11)
5

---

Published as a conference paper at ICLR 2025
Such property is mainly derived based on the standard identity of Gaussian variable:
X ~ N (ux, , ~ N (jx,, , - X + N (ux + py, o + +
(12)
The additive accuracy property makes it feasible to derive the Bayesian flow distribution pr (0
x; i) - Po @ 0g x. Ei= Ot for the simulation-free training of Eq. ( should be noted that the standard identity in Eq. 1TT does not hold in the von Mises distribution. Hence there exists an important difference between the original Bayesian flow defined on Euclidean space and the Bayesian flow of circular data on T based on von Mises distribution. With prior 0 {0, 0}, we could formally represent the non-additive accuracy issue as:
Po (c" 0, 10,;;a 02 ahe an) 5(c Ola (xb) + (0 Po (c" 0', x; 0s))
(yslaa)) BuM(y. 6(c (c- II cos y. +oy cos yo: Ogg sin y. + sin y,
(13)
A more intuitive visualization could be found in Fig. This fundamental difference between periodic Bayesian flow and that of Graves et al. (2023) presents both theoretical and practical challenges, which we will explain and address in the following contents.
Entropy Conditioning As a common practice in generative models (Ho et al. 2020a Lipman et al. 2022 Graves et al. 2023), timestep is widely used distinguish among generation states by feeding the timestep information into the networks. However, this paper shows that for periodic Bayesian flow, the accumulated accuracy c; is more effective than time-based conditioning by informing the network about the entropy and certainty of the states 0,. This stems from the intrinsic non-additio accuracy which makes the receiver's 's accumulated accuracy c not bijective function of t. but distribution conditioned on accumulated accuracies c; instead. Therefore, the entropy parameter e is taken logarithm and fed into the network to describe the entropy of the input corrupted structure. We verify this consideration in Sec. 5.3
Reformulations of BFN. Recall the original update function with Gaussian sender distribution, after receiving noisy samples y1, .2....... Y; with accuracies 11, ........ O₁, the accumulated accuracies of the receiver side could be analytically obtained by the additive property and it is consistent with the sender side. However, previously mentioned, this does not apply to periodic Bayesian flow, and some of the notations in original BFN (Graves et al. 2023) need to be adjusted accordingly. We maintain the notations of sender side's one-step accuracy and added accuracy B, and alter the notation of receiver's accuracy parameter as e, which is needed to be simulated by cascade of Bayesian updates. We emphasize that the receiver's accumulated accuracy e no longer function of t (differently from the Gaussian case), and it becomes distribution conditioned on received accuracies 01, 02 ................................ a, from the sender. Therefore, we represent the Bayesian flow distribution of von Mises distribution as Py (0) x; Q1, 0'2, a;). And the original simulation-fri training with Bayesian flow distribution is no longer applicable this scenario.
Fast Sampling from Equivalent Bayesian Flow Distribution Based on the above reformulations, the Bayesian flow distribution of von Mises distribution is reframed as:
PF 0 x; 01, ......... ,.,) = Ep, (0, [0..x..) ... By,, (0.-1 x.a. Pe (0,10,-1,;;0; x;
(14)
Naively sampling from Eq. (14) requires slow auto-regressive iterated simulation, making training unaffordable. Noticing the mathematical properties of Eqs. and we transform Eq. (1) the equivalent form:
Pr(m: x; 01, 01,,002--0 ..)) = EuM(y, (xly) 1x,0,) atan2
0g COS Yj
2 sin y,))
(15)
pF(ci]x; a1, au,, .2,...,a.)) ax)) = EvM(y, (x.1)...M (x,,s) ō(c;
aj COS
aj sin y,
(1222 (
j=1
(16)
which bypasses the computation of intermediate variables and allows pure tensor operations, with negligible computational overhead.
Proposition 4.1. The probability density function of Bayesian flow distribution defined by Eqs. and 116 is equivalent to the original definition in Eq. (14)
[15

---

Published as a conference paper at ICLR 2025
Numerical Determination of Linear Entropy Sender Accuracy Schedule Original BFN designs
the accuracy schedule (tt) to make the entropy of imput distribution linearly decrease. As for crystal generation task, to ensure information coherence between modalities, we choose sender accuracy
schedule 01; 0.2.-... O; that makes the receiver's ' belief entropy (1,) = H (-1.)) = H (prr(-ee (-)c.) linearly decrease w.r.I. time 4, given the initial and final accuracy parameter c(0) and c(1). (1). Due to the intractability of Eq. (30), we first use numerical binary search in [0, c(1)] to determine the
receiver's c(t;) for i = I..... n by solving the equation H (c(f:)) (1 4.) (c(0) + tH (c(1).
Next, with c(fz), we conduct numerical binary search for each Og in [0, c(1) by solving the equations
Ey-eM(s,..o)f Iy o3 + ci_1 + 20:0;11 cos(y m,-1) = c(t)) (by Eq. (45)) from i = 1 to i = n for arbitrarily selected x E [-T, .T)
After tackling all those issues, have now arrived at BFN architecture for effectively modeling crystals. Such BFN can also be adapted to other type of data located in hyper-torus T
4.2 EQUIVARIANT BAYESIAN FLOW FOR CRYSTAL
With the above Bayesian flow designed for generative modeling of fractional coordinate F, we are able to build equivariant Bayesian flow for each modality of crystal. In this section, we first give an overview of the general training and sampling algorithm of CrysBFN (visualized in Fig. [ Then, we describe the details of the Bayesian flow of every modality. The training and sampling algorithm can be found in Algorithm | and Algorithm 2
Overview Operating in the parameter space Q. = (0A, 0-, 0F), CrysBFN generates high- fighffidllity
crystals through a joint BFN sampling process on the parameter of atom type 04, lattice parameter 02 {u-, p²). }. and the parameter of fractional coordinate matrix 0f {m" cf'). We index the n-steps of the generation process in a discrete manner i, and denote the corresponding continuous notation t; = i/n from prior parameter 0.M to considerably low variance parameter 0. (i.e. large p", m nr. and centered Aa).
At training time, CrysBFN samples time i ~ U [1, and 0,M, from the Bayesian flow distribution of each modality, serving as the input the network. The network v outputs (0 41-1) L ( 1 1
11) and conducts gradient descents loss function Eq. (5) for each modality. After proper training, the sender distribution ps can be approximated by the receiver distribution pn-
At inference time, from predefined O,M, we conduct transitions from MMM, to M.M by: (1) sampling
y; ( Mi t. a;) according to network prediction / (0 ( 411 and (2) performing Bayesian update 1( MO.M..MMM.. o:) for each dimension.
Bayesian Flow of Fractional Coordinate F The distribution of the prior parameter 0f is defined as:
p(05) def {uM 0sxN. Osxx), 8(55" - 0sxN)} = (U 10, - 03xN)}
(17)
Note that this prior distribution of m. is uniform over [0, 1), ensuring the periodic translation invariance property in Definition [ sender and receiver distribution (deduction can be found in Appendix[A.7]
The training objective is minimizing the KL divergence between
(a()) Io(as)
LF n (0Fl) aa) 04
(1 cos F FaenFF-rr(0 F
(18)
where (1(x)) and Ji (4) are the zeroth and the first order of modified Bessel functions. The transition from 0-- to 0. is the Bayesian update distribution based on network prediction:
p( MM =
a 6(0 h(0"I.y,a:))
(19)
Proposition 4.2. With V as periodic translation finiaariantfuncttin namely V o (0A w(O - A 0", 0². + t), Vt R3, the marginal distribution of p(F..) defined by Eqs. (17 and is periodic translation invariant.
t), 0% wi V (0A
Bayesian Flow of Lattice Parameter L Noting the lattice parameter L located in Euclidean space, set prior the parameter of isotoopic multivariate normal distribution of def { pb3
7

---

Published as a conference paper at ICLR 2025
{0ax3: 1ax3) such that the prior distribution of the Markov process μ n² is the Dirac distribution 5(140 - 0) and 8(po 1). which ensures -3))-invae of prior distribution of L. By Eq. 77 from Graves al. (2023). the Bayesian flow distribution of the lattice parameter L is:
( = I()) ((t - -(())
(20)
where y() - of and 01 is the predefined hyper-parameter controlling the variance of input distribution at under linear entropy accuracy schedule. The variance parameter p does not need to be modeled and fed to the network, since it is deterministic given the accuracy schedule. After sampling j from ptz the training objective defined as minimizing KI. divergence between sender and receiver distribution (based on Eq. 96 in Graves et al. (2023))
L 1
n2 1 L1
=
2
11 (1.)) (uf (L)--))
21/n
(21)
where the prediction term (0 ing, the generation process is defined the Bayesian update distribution given network prediction:
i-1: 14-1)
is the lattice parameter part of network output. After train-
plu
1 (mt) ( 1 1;-1), (), 1i
(22)
Proposition 4.3. With V equivvrnnn function namely (OA, OF Q00 t) QV (0A, A OP, 0 t), VQ Q I, the marginal distribution of p(p!)) defined by Eq. (22) 0(3)-
invariant.
Bayesian Flow of Atom Types A Given that atom types are discrete random variables located in a simplex S.K, the prior parameter of A is the discrete uniform distribution over the vocabulary 0 def
K 1IxN- With the notation of the projection from the class index j to the length K one-hot def
vector (ez)s
def jjjk where ej E _K e (e....... Cay) ) RKN, the Bayesian flow distribution
of atom types A is derived in Graves et al. (2023):
(0A A:t)
04
(23)
E cv (0))
where (1) is the predefined accuracy schedule for atom types. Sampling 0²² from P as the training signal, the training objective is the n-step discrete-time loss for discrete variable (Graves et al. 2023):
LA = n.E.u (1,m) (0 11) (Kea-1),a;KK] In (y as (Kea - 1), a1K1)
N
K
Σ d=1
In
0 (4:4-1WWW (1)0) as (Kez 1), a)
(24)
where I E RKNNNN and E RXD, When sampling, the transition from 0.1, to 0:1 is derived as:
p(O,A MM
10 v ( 4--1): (4 4--1)
(25)
The detailed training and sampling algorithm could be found in Algorithm T and Algorithm 2
5 EXPERIMENTS
We evaluate on two crystal generation tasks: ab initio generation in Sec. 5.1 and stable structure prediction task in Sec. 5.2 Ablation studies are detailed in Sec. 5.3 to validate design choices. We
provide the implementation details in Appendix C
Following Xie al. (202)); Jiao al. (2023). we choose the following datasets for evaluation: 1) Perov-5 (Castelli et al. 2012a b) is composed of 18,928 perovskite crystals of similar structures, with atoms unit cell sharing the chemical formula ABX;- 2) Carbon-24 (Pickard, 2020) contains 10,153 crystals with 6-244 atoms in a cell, and all atom types are carbon. 3) MP-20 (Jain et al. 2013) selects 45,231 stable inorganic materials from Material Projects (Jain et al., 2013). inc including the majority of experimentally- veriifiie materials with at most 20 atoms in a unit cell. 4) MPTS-52 (Jia et al. 2023) consists of 40,476 crystals up to 52 atoms per cell, which is a more challenging extension of MP-20. All crystals are reduced as Niggli cells (Niggli 1928). The procedure to split the datasets
into training, validation, and testing subsets adheres to prior practices (Xie et al. 202 Jiao et al. 2023
8

---

Published as a conference paper at ICLR 2025
Table 1: Results on ab initio generation task. Baseline results are from (0223
Xie et
202
Jiao et al.
Miller et al. 2024).
Validity (%)
Coverage (%)
Property dip
Data
Method
Struc.
Comp.
COV-R
COV-P
dp
detem
Peruv-5
Cond-DFC- VAE (Court al
2020
73.60
82.95
73.92
10.13
2.268
4.111
0.8373
G-SchNet (Gebauer et al. 2019
99.92
98.79
0.18
0.23
1.625
4.746
0.0368
P-G-SchNet (Gehater ct al. 2019
79.63
99.13
0.37
0.25
0.2755
1.388
0,4552
CDVAE
el al.
100.0
98.59
99.45
98.46
0.1258
0.0264
0.0628
DitICSP
100.0
98.85
99,74
98.27
0.1110
0.0263
0.0128
CrysBEN
100.0
98.86
99.52
98.63
0.0728
0.0198
0,0098
Carbon-24
G-SchNet (Gebauer al.
2019
99.94
0.00
0.00
0.9427
1.320
P-G-SchNer
Gchauer al.
2019
48.39
0.00
0.00
1.533
134.7
CDVAB
Xie el at
202
100.0
99.80
83.08
0,1407
0,2850
DifCCSP
(1ao el al.
2025
100.0
99,90
97.27
0.0805
0.0820
CrysBEN
100.0
99.90
99.12
0.0612
0,0503
MP-20
G-SchNet (Gebauer et al. 2019
99.65
75.96
38.33
99.57
3.034
42.09
0.6411
P-G-SchNet Gihhaeer et al. 2019
77.51
76.40
41.93
99.74
4.04
2.448
0.6234
CDVAR
et aL.
100.0
86.70
99.15
0.6875
0.2778
1.432
DifICSP Oracs
2045
100.0
83.25
99.71
99.76
0.3502
0.1247
0.3398
FlowMM
(Miler et al.
2024
96.85
83.19
99.49
99.58
0.239
0.083
CrysBEN
100.0
87.51
99.09
99.79
0.2067
0.0632
0.1628
Table 2: Results on stable structure prediction task. Baseline results are from Jiao et al.
2023 Miller
al. 2024).
Perov-5
MP-20
MPTS-57 --2
Match ratef
RMSEJ, Match rate RMSEJ Match ratef
RMSEJ
CDVAE
Xie et al.
2021
45.31
0.1138
33.90
0.1045
5.34
0,2106
DifCCSP
Jiao
2023
52.02
0.0760
51.49
0.0631
12.19
0.1786
FlowMM Miller et al.
2024
53.15
0.0992
61.39
0.0566
17.54
0.1726
CrysBFN
54.69
0.0636
64.35
0.0433
20.52
0.1038
5.1
AB INITIO GENERATION
Baselines For this task, the compared baselines include: two-stage VAE-based methods Cond-
DFC-VAE (Court et al. 2020) and CDVAE (Xie et al. 2021) 2) auto-regressive method G-
SchNet (Gebauer et al. 2019), and its periodic adaptation P-G-SchNet (Xie et al. 2021); 3)
diffusion-based joint generation approach DifTCSP (Jiao et al. 2023). 4) flow-matching-biee
approach FlowMM (Note that FlowMM only reports results on MP-20 and excludes dg). We follow
(Hoogeboom et al. 2022 Jiao al. 2023) to sample atom numbers from distribution that is
pre-computed based on atom numbers in the training dataset. Performance Indicators Following previous work (Xie et al. 2021), , we evaluate the efficacy of our model from three aspects: 1) Validity: Structure and compositional validity of randomly generated 10000 materials. 2) Coverage: Cover- age score between generated 10000 materials and test set, defined by average minimum structure distance and average minimum compositional distance. 3) Property Statistics: the earth mover's distance (EMD) between the property distribution of generated crystals and test dataset crystals. Monitored properties include density (p. unit g/em²), energy predicted by an independent GNN (E, unit e V/atom), and the number of unique elements elem.).
The evaluation metrics for ab initio generation tasks are listed in Tab. I Our method consis- tently achieves better or competitive property statistics and generation precision on three datasets compared to baseline generative models. For compositional metrics including detem and composi- tional validity, our method demonstrates bigger performance improvement for the more challenging dataset MP-20 (+4.34% compared to DiffCSP with the same level of detem), underscoring the importance of modeling atom types in the simplex space.
Results
5.2 STABLE STRUCTURE PREDICTION
In this section, extend method to stable structure prediction task, where the modeling target is p(L, F).. The condition of atom types A is incorporated into the network by concatenating node feature and atom type embedding, following Jiao al. 2023)

---

Published as a conference paper at ICLR 2025
Baselines Following the practices in Jiao et al. (2023) select baselines generative approaches including Diffusion-based approaches CDVAE and DiffCSP and the recent flow- wwwcmathhhnnbbee based method, FlowMM, which only reports results the MP-20 dataset for this task. Performance Indicators The measured performance indicators for this task are Match Rate and RMSD computed by Structur reMa cher class with thresholds stol= .0 5, angle_tol=]]
1tol=0. 3 in pymatgen (Ong et al.) 2013), between the predicted structure candidates and the groundtruth structure given the composition. Results As summarized in Tab. - CrysBFN achieves consistent performance improvement over baseline methods, especially for more challenging datasets (~ 13% higher match rate than DifCCSP for MP-20 and 40% lower RMSE compared FlowMM for MPTS-52).
5.3 ABLATION STUDY
Table 3: Ablation studies on MP-20.
Using MP-20 dataset and stable structure predic- tion task, we validate the necessity of proposed components of CrysBFN with results summa- rized in Tab. 3
Match rate (%)
RMSE
CrysBFN
64.35
0.0433
1) By removing the entropy parameter condi- tion c and using time as condition, match rate drops to 52.16%. proving that, different from original BFN, the non-additive accuracy dynam- ies requires the network to model both mean parameter m and entropy parameter condition
w/o entropy cond. w//o approx. sch. w/o torus BFN
52.16 49.76
0.0631
0.0643
6.17
0,3822
1k Batches Sim. Time (s)
Iterated Sim. Fast Sim.
356.1
92.6
2) By altering the approximated linear-entropy sender accuracy schedule to the hand-designed
roughly linear-entropy schedule c(t) = tc (1), we validate the effect of exact searched linear entropy schedule.
3) By replacing the proposed hyper-torus BFN with the original continuous BFN, we observe poor match rate at 6. 17%, indicating the importance of redesigning BFN for crystal data. Calculating the computational time for simulating 1000 batches, we observe 4x efficiency, verifying fast sampling rate considering the full training procedure of MP-20 requires 150k steps.
5.4 SAMPLING EFFICIENCY EXPERIMENT
We compare the sampling efficiency of Crys- BFN and DiffCSP over the CSP task on the MP-20 dataset, based on the Number of Func- tion Evaluations (NFE), iee, the number of net- work forward passes. The experimental results is plotted in Fig. 4 Notably, CrysBFN achieves remarkable match rate of 60.02% with only 10 step network forwards, surpassing DifiCSP's performance of 51.49% 2000 step network forwards. This illustrates the exceptional sam- pling efficiency of CrysBEN
0.6
0.5
Match Rate
0.4
DiflCSP(12.3M)
CrysBFN(12.3M)
0
500
1000
1500
2000
Number of Network Forwards
Figure 4: Experimental results on MP-20 with different Number of Function Evaluations (NFE) i.e. number of network forward passes.
CONCLUSION
In this paper, we present the first periodic Bayesian flow modeling the hyper-torus, addressing an unprecedented theoretical issue related to non-additive accuracy. Specifically, we introduce a novel entropy conditioning mechanism, theoretical reformulations of BFN, fast sampling algorithm, and numerical method for determining the accuracy schedule. Leveraging the proposed periodic Bayesian flow, we implement the first periodic E(3) equivariant Bayesian flow networks for crystal generation. Our approach achieves state-of of of-the-art performance in crystal generation, with efficiency improved by two orders of magnitude. Additionally, our methodology can be adapted to a wide range of data types and tasks involving hyper-torus data.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
We confirm that work complies with the ICLR Code of Ethics, and we have carefully considered potential ethical concerns related to the development and use our proposed method, CrysBFN, for crystal generation. Our method is designed for general crystal generative modeling tasks and does not involve sensitive data or tasks. We strongly encourage users to ensure compliance with relevant privacy regulations and critically assess the model's ' outputs. We are confirmed that there no conflict interest, financial sources or other factors, that could influence the development or presentation of this work.
With these considerations, we do not anticipate any violations of the ICLR Code of Ethics in the development use of this model. And stress once again that CrysBEN should not be used for malicious purposes, such as creating harmful structures.
REPRODUCIBILITY STATEMENT
To ensure reproducibility, we give a detailed derivation the periodic Bayesian flow in Appendix A and the proof of the propositions in Appendix B All datasets and performance evaluation method used our experiments are publicly available and clearly specified or cited in Sec. We provide our implementation details including training and sampling procedure, prperpparametes, used computational resources and anonymous code repository link in Appendix C
ACKNOWLEDGMENTS
This work supported by the National Science and Technology Major Project (2022ZD01 17502), the Natural Science Foundation of China (Grant No. 62376133, 624061 770) and sponsored by Beijing Nova Program (20240484682) and the Wuxi Research Institute of Applied Technologies, Tsinghua University under Grant 20242001120..
REFERENCES
Mila AI4Science, Alex Hernandez-Garcia. Alexandre Duval, Alexandra Volokhova, Yoshua Bengio, Divya Sharma, Pierre Luc Carrier, Michat Koziarski, and Victor Schmidt. Crystal-gfn: sampling crystals with desirable properties and constraints. arXiv preprint arXiv:2310.044 240.095,, 2023.
Fan Bao, Min Zhao, Zhongkai Hao, Peiyao Li, Chongxuan Li, and Jun Zhu. Equivariant energy- guided SDE for inverse molecular design. In The Eleventh International Conference on Learning Representations, 2023.
James Bergstra, Daniel Yamins, and David Cox. Making a science of model search: Hyperparameter optimization in hundreds dimensions for vision architectures. In International conference on machine learning, pp. 115-123. PMLR, 2013.
Peter E Blöchl. Projector augmented-wave wave method. Physical review B, 3050044):17553 1994.
Valentin De Bortoli, Emile Mathieu, Michael John Hutchinson, James Thornton, Yee Whye Teh, and Arnaud Doucet. Riemannian score-based generative modelling. Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
Tomás Bučko, Jürgen Hafner, and János G Ángyán. Geometry optimization of periodic systems using internal coordinates. The Journal of chemical physics,
2005.
Keith T Butler, Daniel W Davies, Hugh Cartwright, Olexandr Isayev, and Aron Walsh. Machine learning for molecular and materials science. Nature, 559(7715):547-55)) 2018.
Zhendong Cao. Xiaoshan Luo, Jian Lv, and Lei Wang. Space group informed transformer for crystalline materials generation. arXiv preprint arXiv:2403. 15734, 2024.

---

Published as a conference paper at ICLR 2025
Ivano E Castelli, David D Landis, Kristian S Thygesen, Søren Dahl, Ib Chorkendorff, Thomas F Jaramillo, and Karsten W Jacobsen. New cubic perovskites for one-and two-photon water splitting using the computational materials repository. Energy & Environmental Science, 5(10):9034-904 2012a.
Ivano E Castelli, Thomas Olsen, Soumendu Datta, David Landis, Søren Dahl, Kristian S Thygesen, and Karsten W Jacobsen. Computational screening of perovskite metal oxides for optimal solar light capture. Energy & Environmental Science, 5(2):5814 58199 2012b.
Lowik Chanussot, Abhishek Das, Siddharth Goyal, Thibaut Lavril, Muhammed Shuaibi, Morgane Riviere, Kevin Tran, Javier Meeas-Dommnngo, Caleb Ho, Weihua Hu, Aini Palizhati, Anuroop Sriram, Brandon Wood, Junwoong Yoon, Devi Parikh, C. Lawrence Zitnick, and Zachary Ulissi. Open catalyst 2020 (oc20) dataset and community challenges. ACS Catalysis, 2021. doi: 10.1021/ coscatal.0c045255
Chi Chen and Shyue Ping Ong. A universal graph deep learning interatomic potential for the periodic table. Nature Compulational Science, 2(11):718-77) 728, 2022a.
Chi Chen and Shyue Ping Ong. A universal graph deep learning interatomic potential for the periodic table. Nature Computational Science, 2(1 1):718-728, 2022b.
Chi Chen, Weike Ye, Yunxing Zuo, Chen Zheng, and Shyue Ping Ong. Graph networks as universal machine learning framework for molecules and crystals. Chemistry of Materials, 31(9):3564-357) 2019.
Ricky TQ Chen and Yaron Lipman. Riemannian flow matching general geometries. arXiv preprint arXiv:2302.03660 2023.
Guanjian Cheng, Xin-Gao Gong, and Wan-Jian Yin. Crystal structure prediction by combining graph network and optimization algorithm. Nature communications, 13(1):1-8, 2022.
Gabriele Corso, Hannes Stärk, Bowen Jing, Regina Barzilay, and Tommi S. Jaakkola. Diffdock:
Diffusion steps, twists, and turns for molecular docking. In The Eleventh International Conference on Learning Representations, 2023.
Callum J Court, Batuhan Yildirim, Apoorv Jain, and Jacqueline M Cole. 3-d inorganic crystal structure generation and property prediction via representation learning. Journal of chemical information and modeling, 60(10):4518-4533 2020.
Daniel W Davies, Keith T Butler, Adam J Jackson, Jonathan M Skelton, Kazuki Morita, and Aron Walsh. Smact: Semiconducting materials by analogy and chemical theory. Journal of Open Source Software, 4(38):1361, 2019.
Gautam Desiraju. Cryptic crystallography. Nature materials, 1(2):77-79, 2002.
Howard Flack. Chiral and achiral crystal structures. Helvetica Chimica Acta, 86(4):905 2003.
Scott Fredericks, Kevin Parrish, Dean Sayre, and Qiang Zhu. Pyxtal: A python library for crystal structure generation and symmetry analysis. Computer Physics Communications, 261:107810, 2021. ISSN 0010-4655. doi: http:////c....eeee.....////////. sciencedi www.///ciine/.at.cccooooooc
107801000000000000.11100 URL http::/wwww
Fabian Fuchs, Daniel E. Worrall, Volker Fischer, and Max Welling. Se(3)-transformers: 3d roto- translation equivariant attention networks. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florins Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NewrlPS 2020, December 6-12, 2020, virtual, 2020.
Johannes Gasteiger, Shankari Giri, Johannes T. Margraf, and Stephan Günnemann. Fast and uncertainty-awar directional message passing for non-equiibrrum molecules. In Machine Learn- ing for Molecules Workshop, NeurIPS, 2020.
12

---

Published as a conference paper at ICLR 2025
Johannes Gasteiger, Florian Becker, and Stephan Günnemann. Gemnet: Universal directional graph neural networks for molecules. Advances in Neural Information Processing Systems, 34: 6790-6802, 2021a.
Johannes Gasteiger, Florian Becker, and Stephan Günnemann. Gemnet: Universal directional graph neural networks for molecules. In Conference on Neural Information Processing Systems (NeurIPS), 2021b.
Niklas Gebauer, Michael Gastegger, and Kristof Schütt. Symmetry-adapted generation of 3d point sets for the targeted discovery of molecules. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' D'Alccé-BBuc E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 7566-7578. Curran Associates, Inc., 2019.
Niklas WA Gebauer, Michael Gastegger, Stefaan Hessmann, Klaus-Rober Müller, and Kristof Schütt. Inverse design of 3d molecular structures with conditional generative neural networks. Nature communications, 13(1):1-11, 2022.
Justin Gilmer, Samuel Schoenholz, Patrick Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference muchine learning, pp. 1263-1272. PMLR, 2017.
Colin W Glass, Artem Oganov, and Nikolaus Hansen. Uspex evolutionary crystal structure prediction. Computer physics communications, 175(11-12)713-720, 2006.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139-11 2020.
Alex Graves, Rupesh Kumar Srivastava, Timothy Atkinson, and Faustino Gomez. Bayesian flow networks. arXiv preprint arXiv:2308.07037, 2023.
Ralf W Kunnst-eunst,eee, Nicholas K Sauter, and Paul D Adams. Numerically stable algorithms for the computation of reduced unit cells. Acta Crystallographica Section A: Foundations of Crystallography, 60(1):1-6, 2004.
Nate Gruver, Anuroop Sriram, Andrea Madotto, Andrew Gordon Wilson, C Lawrence Zitnick, and Zachary Ulissi. Fine-tuned language models generate stable inorganic materials as text. arXiv preprint arXiv:2402.04333 2024.
Peter Guttorp and Richard A Lockhart. Finding the location of a signal: A bayesian analysis. Journal of the American Statistical Association, 83(402):322-33 -300, 1988.
Jürgen Hafner. Ab-initio simulations of materials using vasp: Density-functional theory and beyond. Journal of computational chemistry, (9(13):04442207) 2008.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020a.
Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020b.
Jordan Hoffmann, Louis Maestrati, Yoshihide Sawada, Jian Tang, Jean Michel Sellier, and Yoshua Bengio. Data-driven approach to encoding and decoding 3-d crystal structures. arXiv preprint 19XX0:1909.00949, 2019.
Detlef WM Hofmann and Joannis Apostolakis. Crystal structure prediction by data mining. Journal of Molecular Structure, 647(1-3):17-99 2003.
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multinomial diffusion: Learning categorical distributions. Advances in Neural Information 2021.
Processing Systems, 34:12454-1244
Emiel Hoogeboom, Victor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant diffusion for molecule generation in 3d. In International Conference on Machine Learning, pp. 8887- PMLR, 2022.
13

---

Published as a conference paper at ICLR 2025
Jianjun Hu, Wenhui Yang, and Edirisuriya M Dilanga Siriwardane. Distance matrix-based crystal structure prediction using evolutionary algorithms. The Journal of Physical Chemistry A, 124(51): 10909-10919, 2020.
Jianjun Hu, Wenhui Yang, Rongzhi Dong, Yuxin Li, Xiang Li, Shaobo Li, and Edirisuriya MD Siri- wardane. Contact map based crystal structure prediction using global optimization. CrystEngComm, 23(8):165-1177. 2021.
Chin-Wei Huang, Milad Aghajohari, Joey Bose, Prakash Panangaden, and Aaron Courville. Rieman- nian diffusion models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
TL Jacobsen, MS Jørgensen, and B Hammer. On-the-fly machine learning of atomic potential in density functional theory structure optimization. Physical review letters, 120(2):026 102, 2018.
Anubhav Jain, Shyue Ping Ong, Geoffroy Hautier, Wei Chen, William Davidson Richards, Stephen Dacek, Shreyas Cholia, Dan Gunter, David Skinner, Gerbrand Ceder, et al. Commentary: The materials project: A materials genome approach to accelerating materials innovation. API. materials, 1(1):01 1002, 2013.
Rui Jiao, Wenbing Huang, Peijia Lin, Jiaqi Han, Pin Chen, Yutong Lu, and Yang Liu. Crystal structure prediction by joint equivariant diffusion on lattices and fractional coordinates. In Workshop on" Machine Learning for Materials' "ICLR 2023, 2023.
Rui Jiao, Wenbing Huang, Yu Liu, Deli Zhao, and Yang Liu. Space group constrained crystal generation. arXiv preprint arXiv:2402.03992, 2024.
Bowen Jing, Gabriele Corso, Jeffrey Chang, Regina Barzilay, and Tommi Jaakkola. Torsional diffusion for molecular conformer generation. Advances in Neural Information Processing Systems, 35:24240-24222 2022.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, Alex Bridgland, Clemens Meyer, Simon A A Kohl, Andrew Ballard, Andrew Cowie, Bernardino Romera-Paredes, Stanislav Nikolov, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David Reiman, Ellen Clancy, Michal Zielinski, Martin Steinegger, Michalina Pacholska, Tamas Berghammer, Sebastian Bodenstein, David Silver, Oriol Vinyals, Andrew W Senior, Koray Kavukcuoglu, Pushmeet Kohli, and Demis Hassabis. Highly accurate protein structure prediction with AlphaFold. Nature, 596 doi:
(7873):583-589, 2021.
Sungwon Kim, Juhwan Noh, Geun Ho Gu, Alan Aapuru-Guzile and Yousung Jung. Generative adversarial networks for crystal structure prediction. ACS central science, 641::1412-1420) 2020.
Thomas N Kipf and Max Welling. Variational graph auto-encoders. arXiv preprint arXiv: 161.00361..07300. 2016.
Toru Kitagawa and Jeff Rowley. von mises-fisher distributions and their statistical divergence. arXiv preprint 200.:2200..0519, 2022.
Astrid Klipfel, Yaël Frégier, Adlane Sayede, and Zied Bouraoui. Unified model for crystalline material generation. arXiv preprint wrXiv:2306.04510, 2023.
Walter Kohn and Lu Jeu Sham. Self-consistent equations including exchange and correlation effects. Physical review, 140(4A):A1 1333, 1965.
Georg Kresse and Jürgen Furthmüller. Efficiency of ab-initio total energy calculations for metals and semiconductors using a plane-wave basis set. Computational materials science, 6(1):15-50, 1996.
Gerhard Kurz, Igor Gilitschenski, and Uwe Hanebeck. Efficient evaluation of the probability density function of a wrapped normal distribution. In 2014 Sensor Data Fusion; Trends, Solutions, Applications (SDF), pp. 1-5. IEEE, 2014.
Minoru Kusaba, Chang Liu, and Ryo Yoshida. Crystal structure prediction with machine learning- based element substitution. Computational Materials Science, 211:11 14111199,, 2022.
14

---

Published as a conference paper at ICLR 2025
Christophe Ley and Thomas Verdebout. Modern directional statistics. Chapman and Hall/CRC, 2017.
Peijia Lin, Pin Chen, Rui Jiao, Qing Mo, Cen Jianhuan, Wenbing Huang. Yang Liu, Dan Huang, and Yutong Lu. Equivariant diffusion for crystal structure prediction. In Forty-first International Conference on Machine Learning, 2024.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.081 02747, 2022.
Yue Liu, Tianlu Zhao, Wangwei Ju, and Siqi Shi. Materials discovery and design using machine learning. Journal of Materiomicss pp. 159-177. Sep 2017. doi: 10. jo..t.66..aaa2211.....0.00 URL jmat 2017 08.002000..2...
http://dx doi. wwt.10.g//00.
6/ www.ddo........... 101
Shitong Luo, Yufeng Su, Xingang Peng, Sheng Wang. Jian Peng, and Jianzhu Ma. Antigen sppecicc antibody design and optimization with diffusion-based generative models for protein structures. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances Neural Information Processing Systems, 2022.
Xiaoshan Luo, Zhenyu Wang, Pengyue Gao, Jian Lv, Yanchao Wang, Changfeng Chen, and Yanming Ma. Deep learning generative model for crystal structure prediction. upj Computational Materials, 10(1):254, 2024a.
Youzhi Luo, Chengkai Liu, and Shuiwang Ji. Towards symmetry- aware generation of periodic materials. Advances in Neural Information Processing Systems, 36, 2024b.
Kanti V Mardia and SAM El-Atoum. Bayesian inference for the von mises-fisher distribution.
Biometrika, 63(1):203-206, 1976.
Kanti Mardia and Peter Jupp. Directional statistics. John Wiley & Sons, 2009.
Benjamin Kurt Miller, Ricky T. Q. Chen, Anuroop Sriram, and Brandon M Wood. FMowMM: Generating materials with Riemannian flow matching. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the l International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 35664 35686. PMLR, 21-27 Jul 2024. URL https::/ html
proceedings www..ppees/....
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models.
In International Conference on Machine Learning, pp. 8162-8177 PMLR, 2021.
Paul Niggli. Krystallographicche und strulturrheeretischee Grundbegriffe, volume 1. Akademische verlagssesslliccaftt mbh, 1928.
Juhwan Noh, Jaehoon Kim, Helge S Stein, Benjamin Sanchez-Lengeling, John M Gregoire, Alan Aspuru-Guzik, and Yousung Jung. Inverse design of solid-state materials via a continuous repre- sentation. Matter, 165):1370-1384, 2019.
Asma Nouira, Nataliya Sokolovska, and Jean-Claude Crivello. Crystalgan: learning to discover crystallographic structures with generative adversarial networks. arXiv preprint arXiv:1810.1120 2018.
Artem R Oganov, Chris Pickard, Qiang Zhu, and Richard J Needs. Structure prediction drives materials discovery. Nature Reviews Materials, 4(5))331-344 2019.
Shyue Ping Ong, William Davidson Richards, Anubhav Jain, Geoffroy Hautier, Michael Kocher, Shreyas Cholia, Dan Gunter, Vincent L Chevrier, Kristin A Persson, and Gerbrand Ceder. Python materials genomics (pymatgen): A robust, open-source python library for materials analysis. Computational Materials Science, 68:314-319, 2013.
Jiayu Peng, Daniel Schwalbe-Koda, Karthik Akkiraju, Tian Xie, Livia Giordano, Yang Yu, C John Eom, Jaclyn R Lunger, Daniel J Zheng, Reshma R Rao, al. Human and machine-centre" designs of molecules and materials for sustainability and decarbonization. Nature Reviews Materials, (12):991-1009, 2022.
15

---

Published as a conference paper at ICLR 2025
John P Perdew, Kieron Burke, and Matthias Ernzerhof. Generalized gradient approximation made simple. Physical review letters, 7711)3386 1996.
Chris J. Pickard. Airss data for carbon at 10gpa and the c+n+h++ system at Igpa, 2020.
Chris Pickard and RJ Needs. Ab initio random structure searching. Journal of Physics: Condensed Matter, 23(5):053201, 2011.
Evgeny V Podryabinkin, Evgeny V Tikhonov, Alexander V Shapeev, and Artem R Oganov. Acceler- ating crystal structure prediction by machine-learning interatomic potentials with active learning. Physical Review B, 99(6):064114, 14, 2019.
Yanru Qu, Keyue Qiu, Yuxuan Song, Jingjing Gong, Jiawei Han, Mingyue Zheng, Hao Zhou, and Wei-Ying Ma. Molcraft: Structure-based baseure-baaed drug design in continuous parameter space. arXiv preprint Xiv::2404 12141, 2024.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9 2019.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint (xii:2204.006255, 2022.
Zekun Ren, Siyu Isaac Parker Tian, Juhwan Noh, Felipe Oviedo, Guangzong Xing, Jiali Li, Qiao- hao Liang, Ruiming Zhu, Armin G. Aberle, Shijing Sun, Xiaonan Wang. Yi Liu, Qianxiao Li, Senthilnath Jayavelu, Kedar Hippalgaonkar, Yousung Jung, and Tonio Buonassisi. An invert- ible crystallographic representation for general inverse design of inorganic crystals with targeted properties. Matter, 2021. ISSN 2590-2385. doi: https://doi.org/10.. m0..
Hannes Risken and Hannes Risken. Fokker-planck equation. Springer, 1996.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models, 2021.
Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. (n) equivariant graph neural networks. In International Conference on Machine Learning, pp. 9323-9332. PMLR, 2021.
Jonathan Schmidt, Mário RG Marques, Silvana Botti, and Miguel AL Marques. Recent advances and applications of machine learning in solid-state materials science. npj computational materials, 5 (1):83, 2019.
Kristof T Schütt, Huziel E Sauceda, P-J Kindermans, Alexandre Tkatchenko, and K-R Müller. Schnet-a deep learning architecture for molecules and materials. The Journal of Chemical Physics, 148(24):241722,, 2018.
Chence Shi, Shitong Luo, Minkai Xu, and Jian Tang. Learning gradient fields for molecular conformation generation. In International Conference on Machine Learning, pp. 9558-55885555 9568. PMLR, 2021.
Chence Shi, Chuanrui Wang. Jiarui Lu, Bozitao Zhong, and Jian Tang. Protein sequence and structure co-design with equivariant translation. arXiv preprint 10X.v:2210.08761, 2022.
Anshuman Sinha, Shuyi Jia, and Victor Fung. Representation-spaace diffusion models for generating periodic materials. arXiv preprint 0000:2408.00213.. 2024.
Jascha Sohl-Dicke Dal--ickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference Machine Learning, pp. 1256-2265. PMLR, 2015.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yang Song and Stefano Ermon. Improved techniques for training score- booee generative models. Advances in neural information processing systems, -4438-14448, 2020.
16

---

Published as a conference paper at ICLR 2025
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456. 2020a.
Yang Song. Jascha bohl-Dickstein, DiederikP. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. Cornell University arXiv, Cornell University arXiv, Nov 2020b.
Yuxuan Song, Jingjing Gong, Hao Zhou, Mingyue Zheng, Jingjing Liu, and Wei-Ying Ma. Unified generative modeling of 3d molecules with bayesian flow networks. In The Twelfth International Conference Learning Representations, 2023.
Yuxuan Song, Jingjing Gong, Minkai Xu, Ziyao Cao, Yanyan Lan, Stefano Ermon, Hao Zhou, and Wei-Ying Ma. Equivariant flow matching with hybrid probability transport for 3d molecule generation. Advances in Neural Information Processing Systems, 36, 2024.
Julian Straub. Bayesian inference with the von-mises-fisher distribution in 3d, 2017.
Philipp Thölke and Gianni De Fabritiis. Equivariant transformers for neural network based molecular potentials. International Conference on Learning Representations, 2021.
Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-and translatooneqqiivarinn neural networks for 3d point clouds. arXiv preprint arXiv:1802.08213 2018.
Richard Tran, Janice Lan, Muhammed Shuaibi, Siddharth Goyal, Brandon M Wood, Abhishek Das. Javier Heras-Domingo Adeesh Kolluru, Ammar Rizvi, Nima Shoghi, et al. The open catalyst 2022 (oc22) dataset and challenges for oxide electrocatalysis. arXiv preprint arXiv:2206. 08917, 2022.
Brian L. Trippe, Jason Yim, Doug Tischer, David Baker, Tamara Broderick, Regina Barzilay, and Tommi S. Jaakkola. Diffusion probabilistic modeling protein backbones in 3d for the motif- scaffolding problem. In The Eleventh International Conference on Learning Representations, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Clement Vignac, Igor Krawezuk, Antoine Siraudin, Bohan Wang, Volkan Cevher, and Pascal Frossard. Digress: Discrete denoising diffusion for graph generation. In The Eleventh International Confer- ence on Learning Representations, 2023.
Hanchen Wang, Tianfan Fu, Yuanqi Du, Wenhao Gao, Kexin Huang, Ziming Liu, Payal Chandak, Shengchao Liu, Peter Van Katwyk, Andreea Deac, al. Scientific discovery the age of artificial intelligence. Nature, 620(7972):47-660 2023.
Yanchao Wang, Jian Lv, Li Zhu, and Yanming Ma. Crystal structure prediction via particle-swarm optimization. Physical Review B, 82(9):094116, 2010.
Logan Ward, Ankit Agrawal, Alok Choudhary, and Christopher Wolverton. A general-purpose machine learning framework for predicting properties of inorganic materials. npj Computational Materials, 2(1): 1-2, 2016.
Tian Xie and Jeffrey C. Grossman. Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties. Phys. Rev. Lett., 120:145301, Apr 2018. doi: 10.1 103/PhysRevl.ett. 1 105301...
Tian Xie, Xiang Fu, Octavian -uggen Ganea, Regina Barzilay, and Tommi Jaakkola. Crystal diffusion variational autoencoder for periodic material generation. arXiv preprint 10Xiv:21 2021.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural 2018.
networks? arXiv preprint
17

---

Published as a conference paper at ICLR 2025
Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang. Geodiff: A geometric diffusion model for molecular conformation generation. International Conference on Learning Representations, 2021.
Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang. Geodiff: A geometric diffusion model for molecular conformation generation. arXiv preprint arXiv:2203.0292 2022.
Minkai Xu, Alexander S Powers, Ron O Dror, Stefano Ermon, and Jure Leskovec. Geometric latent diffusion models for 3d molecule generation. International Conference on Machine Learning. pp. 38592-38610. PMLR, 2023.
Tomoki Yamashita, Nobuya Sato, Hiori Kino, Takashi Miyake, Koji Tsuda, and Tamio Oguchi. Crystal structure prediction accelerated by bayesian optimization. Physical Review Materials, 2(1): 013803, 2018.
Kegiang Yan. Yi Liu, Yuchao Lin, and Shuiwang Ji. Periodic graph transformers for crystal material property prediction. In The 36th Annual Conference on Neural Information Processing Systems, 2022.
Mengjiao Yang, KwangHwan Cho, Amil Merchant, Pieter Abbeel, Dale Schuurmans, Igor Mor- datch, and Ekin Dogus Cubuk. Scalable diffusion for materials generation. arXiv preprint arXiv:2311.09235, 2023.
Wenhui Yang, Edirisuriya M Dilanga Siriwardane, Rongzhi Dong, Yuxin Li, and Jianjun Hu. Crystal structure prediction of materials with high symmetry using differential evolution. Journal of Physics: Condensed Matter, 33(45)455590, 2021.
Zhenpeng Yao, Benjamín Sänchez-Lengeling N Scott Bobbitt, Benjamin J Bucior, Sai Govind Hari Kumar, Sean Collins, Thomas Burns, Tom Woo, Omar Farha, Randall Q Snurr, et al. Inverse design of nanoporous crystalline reticular materials with deep generative models. Nature Machine Intelligence, 3(1):76 86, 2021.
Sheheryar Zaidi, Michael Schaarschmidt, James Martens, Hyunjik Kim, Yee Whye Teh, Alvaro Sanchez-Gonzalez, Peter Battaglia, Razvan Pascanu, and Jonathan Godwin. Pre-training via denoising for molecular property prediction. arXiv preprint arXiv:2206.00/33 2022.
Claudio Zeni, Robert Pinsler, Daniel Zügner, Andrew Fowler, Matthew Horton, Xiang Fu, Sasha Shysheya, Jonathan Crabbé, Lixin Sun. Jake Smith, et al. Mattergen: a generative model for inorganic materials design. arXiv preprint arXiv:2312.03Rm 0368777 2023.
Xuan Zhang, Limei Wang. Jacob Helwig, Youzhi Luo, Cong Fu, Yaochen Xie, Meng Liu, Yuchao Lin, Zhao Xu, Keqiang Yan, et al. Artificial intelligence for science in quantum, atomistic, and continuum systems. arXiv preprint arXh::2307.0842. 2023.
Yaolong Zhang, Ce Hu, and Bin Jiang. Embedded atom neural network potentials: Efficient and accurate machine learning with physically inspired representation. The journal of physical chemistry letters, 10(17):4962-496 2019.
Yunwei Zhang, Hui Wang, Yanchao Wang, Lijun Zhang, and Yanming Ma. Conputer-assisted inverse design of inorganic electrides. Physical Review X, 7(1):011017, 2017.
Min Zhao, Fan Bao, Chongxuan Li, and Jun Zhu. EGSDE: Unpaired image-to-ime tomee translation via energy-guided stochastic differential equations. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
Nils ER Zimmermann and Anubhav Jain. Local structure order parameters and site fingerprints for quantification of coordination environment and crystal structure similarity. RSC advances, 10(10): 6063-6081, 2020.
18

---

Published as a conference paper at ICLR 2025
π m 2
π
π
π
c. 0
m = C = 5
2
m
2
= 50
m =
2 2. = +00
π m 2 5
m 0,c
m CC
==
Figure 5: Depiction of von Mises distributions with different directions parameters m and concen- tration parameters c. The parameter m denotes the central location about which the distribution is centered, while functions distribution is uniform on the circle. As e increases, the distribution becomes more concentrated around the value m, with serving measure of this concentration. In the limit +oc, the distribution converges to (mm) a Dirac delta distribution centered at m.
measure of the distribution's 's concentration. When 0, the
A BAYESIAN FLOW NETWORKS FOR CIRCULAR DATA
In this section, we provide a detailed derivation of Bayesian flow networks considering periodicity.
A.1 CIRCULAR DATA AND VON MISES DISTRIBUTION
One-dimensional circular data x refers to observations random variables supporting on the circum-
ference of the unit circle defined in directional statistics (Mardia & Jupp 2009 Ley & Verdebout 2017).
This space can be represented by the torus:
T del {z E R2
||2|| = 1}
(26)
For n-diimensional data æ E R", the set of x with every dimension located in T form a compact Riemannian manifold named hyper-torus T" formally.
Wrapped normal distribution used in Jiao et al. (203)) and von Mises distribution used in this paper both circular distributions defined in this space, the probability density function of von Mises distribution with mean direction parameter m and concentration parameter c is
[alm, c) uM(slm exp(c cos(æ: - m))
(27)
2x Io()
where Io(c) is the modified Bessel function of the first kind of order 0 as the normalizing constant. The parameters m and 1/c are analogous to mean j and variance o² in the normal distribution: 1) represents the central location around which the distribution is clustered, while serves as measure of concentration. 2) We give depiction of von Mises distributions with different directions parameters in Fig. 5 When equals zero, the distribution is uniform. As becomes large, the distribution becomes tightly concentrated around the value m, with e quantifying this concentration. In the limit as - +oo, the distribution becomes Dirac delta distribution centered at Its support can be chosen as any interval of length 2x and we in this paper choose [-T, π). Note that the fractional coordinate can be transformed to this interval easily by a linear transformation
g(x) 2 म. For this modeled interval, the map function from R to [- T) is
W|- (x) 7)%2m
x
kmk. 3K E Z
(28)
19

---

Published as a conference paper at ICLR 2025
Such map function is equivalent the map function w(2) [a] kk Z used in Jiao et al. (2023) we choose the modelled interval as [o, 1). Then, we can prove that the probability density distribution of von Mises distribution is equivariant to the periodic translation transformation:
vt E IR, f( + t) x.= (m + t), c) f(x + 2 m + 2 t,
exp(ecos(2 + 2 (m + 2 + ()))
2 2x
exp(ccos((x m))
2T
c))
(29)
The differential entropy of von Mises distribution with mean direction parameter m and concentration parameter c is
[(()
(() (o((c + In (n To(c] (Mardia & Jupp 2009
(30)
(c(c)
We opt for von Mises distribution rather than the wrapped normal distribution used in Jiao al. (2023); [Jig et al. (2022) mainly because of the Bayesian conjugacy of von Mises distribution the posterior of which is conjugate to the prior distribution if the likelihood is parameterized as von Mises distribution which is the fundamental basis of constructing a Bayesian flow. Interestingly, there is an intriguing connection between von Mises distribution and crystal force field that the von Mises distribution is the stationary distribution of drift and diffusion process the circle in harmonic potential corresponding to the harmonic force field of crystals (Risken & Risken 1996).
A.2 INPUT DISTRIBUTION pr( (0) AND SENDER DISTRIBUTION ps (z)
For circular data which locates in quotient space T 3x3 /1- we define the input distribution of the Bayesian Flow Networks as independently factorized von Mises distribution over the interval n, T).
def
m,, cj
(31)
Pr (x nT (s (m()) (
(32)
where (d) [- R) and ((d) [0, oc).
In this paper, to ensuring the periodic translational invariance, the prior parameter of CrysBEN's Bayesian flow is chosed
p(0))) def
(m OSxN, 03xN), 5(co - 0sxx)) = (U(O, - 03xx))
(33)
where 0 is the length of D vector whose entries are all (). Note that this input prior 0g defines multivariate uniform distribution
(x = M(, 2 T
(34)
which ensures periodic E(3) invariance of the prior distribution.
The sender space y is identical to the data space for circular data, And the sender distribution is von Mises distribution centered on æ with concentration parameter a represented by
ps(y|x; a) = IT - (y) (1)) a) == uM(ylx,a)
(35)
A.3 BAYESIAN UPDATE FUNCTION a( AND BAYESIAN UPDATE DISTRIBUTION (- a))
For the receiver, given his last univariate belief parameterized by von Mises distribution with parameter 0,-1 (M4-1, , 1-1), ). he now observes sample y from sender distribution with unknown
20

---

Published as a conference paper at ICLR 2025
and known Now, by Bayesian theorem,
pla (00-1,,-111 plyl:; ( C--1)
(36)
p(y)
(37)
(38)
exp(o y) c-1 cos(x m m+-1)
(39)
The last expression has the form of a von Mises distribution and hence:
p(zly; a, m;-1. = (2;
(40)
where
m, = atana(a sin + c--1 sin m, cos c cos m-11)
(41)
G q= -=va a² 1 + 2ac;11 cos(y m,-1)
(42)
We refer readers interested in more detailed deduction to Mardia & El-Atoum (176));
Guttorp &
Lockhart 1988). Defining the notation for scaler x in circular space as ø del two expressions will be much simpler, intuitive and more similar to the Gaussian form:
[cos x, sin x]] these
h((m;-1,d ct ÿ, a) = {rin c
(43)
where
m. oÿ + G--1m
(44)
Gi
G = q=Ila- + 4- 1
(45)
The Bayesian update distribution Po (- x:)) could be obtained by marginalizing y:
Pu (f' x: ax) = (ps( 6('' h(0, y. a)) = 600 - h(0.y,a))
(46)
A.4 Non-Additive ACCURACY ISSUE
Although all cases including continuous and discrete data proven to enjoy the so-called additive accuracy property considered in Graves et al. (2023) defined as:
Ps (0" 0, x; + on) = Ep (0' e.x.o., Pr (0" 0', on).
(47)
this property does not hold for von Mises distribution. The untenability of this property for von Mises distribution be checked out by considering two steps Bayesian updates with prior 0 (0,0). aa, C'. Vo.
Po (c" 0 x: aa + ab) 8(c aa -as)
(48)
AE (0110.x;a.))) (c" 0', x; or)) = BuM(y|xxa, By I10 Na +op yo II)
(49)
Consequently, the Bayesian flow distribution does not equal one-step Bayesian update distribution with (t)):
Pr (0 x;t) # P.: (0 0,, B()))
(50)
With an accuracy schedule a1 .2..... ... On and B(t) Ej1 aj. this untenability will cause the incongruity between sender's accumulated accuracy (t) and the confidence of receiver's belief e; over his location parameter m,, Hence should differentiate the sender's accuracy schedule and the receiver's 's belief confidencec; And e; is no longer a function but a distribution over t, In consequence,
should define the Bayesian flow distribution parameterized by received sender's accuracies a,, Q2,.... ., ag rather than t. Furthermore, the information of receiver confidence c; should be part of the network input well.
21

---

Published as a conference paper at ICLR 2025
A.5 BAYESIAN FLOW DISTRIBUTION 0= 02, ,.s) AND SENDER ACCURACY SCHEDULE a;
pr(O) x; a (22)..,, 04) Py (0, (o.x:a)) ... Pp (010._1.xxo.)PP
a;) (51)
The original definition of Bayesian flow distribution in Eq. (51) provides iterative algorithm to sample from py but which practically is slow resulting the training unaffordable. In fact, noticing the "additive" property of c,r, by Eqs. 44 and (45). we can sample from pF without iteration:
pr(m) x; (2,..,.......................... 0i) : Euaiy (x,00) ... I (mm atan2
Og COS Vos
Σ
ar sin y:))
(52)
pr(c|x501,00 9.........
0:) By
(x,a)) ( Bemry x 04 CO8 Y
og sin y, 1ll)
(53)
Eqs. (52) and 53) provides an algorithm allowing sampling from PF by pure tensor operations without simulating the flow iteratively. Next, we can define the entropy of the receiver's belief as (t):
(t) def Bpr 02
H (pp((110)
(54)
11 (c;) E (ed (el a2.1 C4 To(cs)
+I((2//((ci)] where i nl
(55)
To ensure the information coherence between modalities, we choose find sender accuracy schedule to make the receiver's 's belief entropy (L) linearly decrease with predefined on Formally, we would like find an accuracy schedule a such that
(C In(2xT((c..] ((t) = (1- + +H 2n Cn Io(c,))
(56)
Note that Eq. 566 can be solved analytically but we can solve numerically by firstly getting the target sender's s accumulated accuracy ((t) via binary search due to the monotonicity of Eq. (30).
Next, we could iteratively search o; from to = n by matching average accuracy toward e; This process could be done only once and the resultant can be cached for each pre-confirmed hyper-parameter (Cns ni) pair.
A.6 OUTPUT DISTRIBUTION Po 00 AND RECEIVER DISTRIBUTION p 0;
Given samples 0 {m, c} from Bayesian flow distribution input, the receiver uses network output L (6, t) to rebuild his belief over ground truth æ termed output distribution. Following Graves al. 2023). parameterize Po using x(0,1) () (0, 1) to be prediction of x:
px 1) 5(x x(),
(57)
Therefore, the receiver distribution is: PR(y a; ()
= xM(()8(0,1).a)
(58)
A.7 DISCRETE-TIME Loss [x"(x)
From Kitagawa & Rowley 2022). the KL divergence between u.M (.M, c1) and vM um 02) is
DKI(eM c1) eM (m2, o2)) In Io(c1) [(a) (cim c2 m'
(59)
Io(c2) Io(ci)
From Eq. (5), the discrete-time loss for circular data
L n (x) nEE,.U (0/xsa a an...
Dr (ps (- x;);; || PP (- 0;
(60)
(as) as(1 - cos(x x(0(-1,t(1))
-nE,, (1,a) py (0 Io(as)
(61)
Continuous- time loss is not tractable because the Bayesian flow distribution is not analytical due to the non-additive accuracy property.
22

---

Published as a conference paper at ICLR 2025
B PROOF OF PROPOSITIONS
In this section, we first prove the crystal geometric invariance of CrysBEN. Crystals remain the same under transformations including permutation, orthogonal transformation, and periodic translation defined as follows:
Definition 1 (Permutation Invariance (Jiao et al. 2023 For any permutation matrix P. PLL, F, A) pp( L, FP, AP), i.e., changing the order of atoms will not change the distribution.
Definition 2 (O(3) Invariance (Jiao et 2023)). For any orthogonal transformation Q E RR3x3 satisfying Q' Q' Q p0 pQQ,, F, A) pl L, F, A), namely, any rotation/reflection of L keeps the distribution unchanged.
2023 ). For any translation t E R Definition 3 (Periodic Translation Invariance (Jiao et al.
10,1)xxN returns the fractional part of each element in F, and 1 E R3x1 is a vector with all elements set to one. It explains that any periodic translation of will not change the distribution.
p(L, w F + + ), = p(L, F. A), where the function w(F) -
[FF]
The combination the above invariances is abbreviated in a compact manner termed periodic E(3) invariance proposed by Jiao al. (2023). The permutational invariance be easily achieved by using GNN frameworks. Periodic translation and rotation are both space group transformations. We first introduce the basic concept of G-invarians.
Definition 4. We call a distribution p(æ) G-invariant if for any transformation g in the group G, plg x) = p(z), and a conditional distribution p(z/(c) is G-equivariant if p(g xlg. = p(æ(c), Vg G.
With lemma from Xu et al. (2021). we can prove a Markov-process-ennete ge-e-peeees--gerreeeeed distribution G- invariant by proving the G-invariance of the prior distribution and the G-equivariance of every transition kernel.
Lemma Xu et al. 2021)). Consider the generation Markov process p(On) [ p(o p(0 Ho d0 If the prior distribution p(O)) is G-invariant and the Markov transitions p(0++1 (.), < are G-equivariant, the marginal distribution p(On) is also G-invariant.
Proof.
Ag E G, p(g 0m) p(g 06) / p(0m.a (ofma110)dde 100)
plg 0o)
II t=0
Plo plg 0c+1lg 0c)d01.
-((0)) /
II p(g Pe+ilg (o..0r)dd 0z)d0
n-1 p(06) /
II p(0e+1/0 )d0 t=
p(06)) / ((0u))((naa00)dl)
= =p(On)-
Therefore, the marginal distribution p(O.) is G-invariant.
With Lemma
I we can prove the following propositions mentioned in the main text:
Proposition 4.3. With V -quivvaaun function namely ( OF Q0" .)
Q i. (0 0", 0². t), VQT Q = I. the marginal distribution of p(u)) defined by Eq. (22 is 0(3)-
invariant.
23

---

Published as a conference paper at ICLR 2025
Proof. The prior is 0(3) invariant since p(p)) δ(μ 0) = - 0),
The transition probability
" (
n (Q Q 111-1--11
=N(QU
I)
Pi
Pf
QQ (1
=N(QU!
I)
(by equivariance of (
Pi
Pi
=N(QUt
Q
a (
1r
11-1Pt-1
a pg
I)
Pi
=N(H aい (H 11 + ul-
1
a I)
(by property of isotropic Normal p.d.f)
Pr
p
=p (ut pet (
=p(u [1 0-, 1: 0f
is equivariant. By Lemma
1 the marginal distribution p(p is O(3)-invariant.
Proposition 4.2. With F as a periodic translation equivariant functiooo namely V F (0A, w(O t), 0z .t, w( (OA O",, f + t), vt R3, the marginal distribution of p(F,) defined by Eqs. and is periodic translation invariant.
Proof. We first prove that the Bayesian update is periodic translational equivariant. Based on Eqs. 44 and (45), we can interpret the Bayesian update of 1, G-13 observing ý with a, the vector addition between ri;-1 and ý with weight G and a. And the periodic translation for x corresponds to the rotation of i with angle
[cos(ı + L 2 + L + 2Tk = [cos(s + sin(f + 1))T
=[cos cos ( sin sin I,, sin cos + cos sin 1)"
coos
sin / cos t
[cos x sin
R
sin t
where R. is the 2-dimensional rotation matrix with angle f. Due the rotational equivariance of 2D vector addition, we can infer that the Bayesian update function h is periodic translational equivariant:
A(tu(m;-1+t)) + + 4- 1., w(y + t), a)) = heni)=(((RRiee h( R.ú. a)
aRR + G: R,m,,
=1
|aRl + c- 1
|aRiÿ + +
aRiÿ G-1
|lar -
||ar + 1-
(Rm, ci) [w(m, c)
(62)
The prior is periodic translation invariant because m U(,,1)
We prove that the Bayesian update distribution pu (m, |m of fff_ (mmm_mmm))))))) (): is periodic transla-
24

---

Published as a conference paper at ICLR 2025
tion equivariant if F is periodic translation equivariant:
po(ur(m + ()) t), cf-1, v (w(mm_++t)):00
(u ( + +t)
+ +)
(m + t) h(w(m, t), +)) y. (by equiv.
=BeM
(m + h(w + و wly + a))
-BeM jr (m M. ( : 1) - h(w(m 1), - w 1), (by Eq.
-BuM (vf (m-_ (m 1o + t) w(h(m - + (by Eq. 66
=BuM(y /v (ma (m
1. equivariance of 6 function) -po(m) [m of wp (mf_))aa)
Next, we prove the following proposition:
Proposition 4.1. The probability density function of Bayesian distribution defined by Eqs. 15 and 166 is equivalent to the original definition in Eq. (14)
Proof. Combining Eqs.
and 45
m,c. a.ÿ: + c
a.ÿ; + 04- 04-10:-11 4-2
o.iii + ... + 0k 0199 + corino
o,yy
V
sin y,
Taking the 2-norm to each side,
m
||2 =112
sin y, 1
y
f=1
j=1
C; = -112 Og COS y
Σ
as sin 111
j=l
The vector direction of m is irrelevant to the scaler c, Therefore,
m
atan2
a4 cos
cos y,
a sin y,)
(63)
-=1
Hence,
pF(m/ a1 .2.............................. .2.....,.))
=Bp, (0:100x000)) .Bp, 10, 100-110,-000 1000-11,-22000
2,300 P) ac))
=ByM(y, (x.e)) ...
atan2
Oi CO8 y
o sin y,)
pr(ci|xe (0100,.00)))))) ... Ep, 101- (m. Omi,,_
-EuM(y, (x.)).. (on)...EEMMyy IEosso..xx. x(c))((c -112
a sin (
ag cos y,,
1=1
1=1
25

---

Published as a conference paper at ICLR 2025
C IMPLEMENTATION DETAILS
Training and Sampling Procedure We provide the training and sampling procedure in Algorithm T and in Algorithm 2
Network Architecture We use CSPNet proposed by Diao (2023) with minor modifications: (1) We add residual connection from the input to the output of fractional coordinates ensuring the equivariance of the network:
F (0, 0 0.f t.) (h, (
(64)
By the periodic translational invariance of (h:; (5)) proved by Jiao et al. (2023), the equivariance of i p (0,,, 4.) can be easily checked:
v (0, w(O + 1),00,,,,,, ww (5)) w0 + +1)
=w(w( (h; (5) + 0.
=w(V- ( 0.. 0 (4) +
(2) We alter the frequency of the Fourier transformation features to model the interval [-T,, with length 2m. (3)The concentration parameter of each fractional coordinate cf is taken logarithm, normalized and concatenated to time embedding, The network aaper-prrameters follows the setting Jiao et (2023) including the number of hidden states and layers.
syper-paarametes. For the network, the CSPNet has layers, 512 hidden states, 128 frequencies for the Fourier feature for each task and dataset following (Tiao et al. 2023). For BFN hyper-paramettrs,
we set of = 0.001 for continuous variable generation, 31 = 1000 for circular variables generation across all datasets and tasks. For discrete variables, 31 = 3.0 for the Perov-5 dataset. The number of steps is searched in (50, 100, 500, 1000, 2000). For optimizations, we apply an AdamW optimizer with an initial learning rate x 10-3 and plateau scheduler with decaying factor of 0.6, patience of 100 epochs, and minimal learning rate I x 10 The weight of every loss 5 x 10-2, The network is trained for 4000, 5000, 1500, and 1000 epochs for Perov-5, Carbon-24, MP-20, and MPTS-52 respectively.
set 31 0.4 for the MP-20 dataset and
Computational Resources All training experiments are conducted on server with NVidia RTX 3090 GPU, 64 x Intel Xeon Platinum 8362 CPU and 256GB memory. Each training task requires one GPU. We also report the required GPU hour across methods to converge in our experimental environment in Tab. 4
Table 4: Comparison of GPU hours required for training across different methods.
GPU Hour
Perov-5 MP-20
MPTS-52
DifICSP Jiao et al. 2023)
8.59
92.22
10.42
FlowMM Miller el al. (2024
16.36
106.37
16.49
CrysBFN
10.19
85.71
12.31
D
MORE RESULTS
Visualizations Here we give visualizations of the ab-initio generated structures from CrysBFN and DiffCSP in Fig. 66 We also provide gif animation of the generation process in our code repository
https://g thub. com/ wu wa- LiinCCrysBi
Error Bars We report the error bar for the crystal structure prediction task following Diao et al. (2023) Tab. running three experiments with different random seeds. The results are similar Tab. 2
Uniqueness, Novelty, and Stability Here we compare the uniqueness, novelty, and stability of ab initio generated samples across methods on MP-20. Using Structure-Matcher in pymatgen with default parameters, generated crystal is considered: 1) unique if it does not match any other generated samples; and 2) novel does not match any crystals in the training set, following prior practices (Zeni al. 2023 Miller et al.) 2024). The stability evaluation procedure follows the approach in Gruver et al. (202)). Finally, a sample considered stable, unique, and novel (S.U.N.) if it satisfies all three conditions. The results are reported in Tab.
26

---

Published as a conference paper at ICLR 2025
DinCC
CrysBEN
Perov-5
Carbon-24
MP-20
Figure Visualizations comparison of the ab-initio generated structures from CrysBEN and DiffCSP.
Table 5: Results on Perov-5 and MP-20 with error bars.
Perov-5
MP-20
Match rate (%)t
RMSFL
Match rate (%)
RMSEL
CDVAE
Xie et al.
2021
05.31
0.1123:10.0026
33.93:10.15
0.1069:10.0018
DifCCSP Jiao et al.
2023)
02.35:10.26
0.0778:10.0030
51.89:10.30
0.0611::0.0015
CrysBFN
54.58:10.13
0.0691+-0.0011
64.33-40.24
0.0445:10.0010
E SAMPLING EFFICIENCY COMPARISON TO ODE SAMPLERS
0.6
Match Rate (%)
0.5
0.4
0.3
DiffCSP(12.3M)
CrysBFN(12.3M)
0.2
FlowMM(28.3M)
500
1000
1500
2000
Number of Network Forwards
Figure 7: Experimental results on MP-20 with different Number of Function Evaluations (NFE) i.e. number of network forward passes including FlowMM. Note that FlowMM has larger parameter size resulting in a less fair comparison.
We present the comparison among them in Fig. 7 and find that FlowMM fails extremely small NFE (20 steps) settings with only 16. 18% match rate, while CrysBFN enjoys 60.02% match rate with 10 steps and consistently achieves the best sampling quality.
F DETAILED DISCUSSION OF RELATED WORKS
Discovering new functional materials has been long-standing scientific problem. Recently, data- driven approaches have been seen as promising solution to address this challenge Peng et al.) (2022).
Two-stage crystal generation methods based on implicit erystal representations One line of approaches indirectly generates crystals in the implicit representation space. Prior practices in-
27

---

Published as a conference paper at ICLR 2025
Table Uniqueness, novelty, and stability experimental results comparison on ab initio generation task on MP-20 dataset.
Method
Unique / %
Novel / % Metastable /
Stable /
S.U.N. Rate / %
DifCCSP (Jiao et al.
2023
96.11
90.95
37.91
12.16
9.44
FlowMM Miileer et
2024
94.79
91.63
32.77
9.23
8,31
CrysBFN
95.29
92.37
45.91
15.82
12.16
Table 7: Ablation study results of entropy-coonditoonig mechanism across datasets.
Perov-5
MP-20
MPTS-52
Match rate
RMSEL
Match ratef RMSEL Match ratef RMSE
w/o entropy conditioning
51.33
0.0753 0.0636
52.16 64.35
0.0631
13,41 20.52
0.1547 0.1038
CrysBFN
54.69
0.0433
cludes transforming crystals into human-designed fingerprint FTCP (Ren et al. 202T), 3D voxels images (Hoffmann et al. 2019). 2D images (Noh et al. 2019). ) 3D bectron-ddesity maps (Court al. 2020p, video-like representation (Yang et al. 2023). embedded atom density (Zhang al. 2019) in StructRepDiff' (Sinha et al.) 2024). However, their generation quality is hampered by the encoding and reconstruction processes, which may not be fully reversible or fail to respect physical symmetries such as rotational and translational invariance. For example, 3D voxel grids (Hoffmann jct al., 2019) and 3D density maps (Court et al. 2020) are invariant to periodic transformations but not B(3 transformations (Zhang al. 2023). and the video-like representation (Yang et al. 2023 is not invariant to permutation, rotation, translation, and periodic transformations.
Direct crystal generation methods Direct material generation in sample space could bypass the above reversibility problem. Prior works (Nouira et al. [2018] Kim et al.) (020)) employ Generative Adversarial Networks (Goodfellow al. 2020) generate crystal structures, while their methods fail to respect crystal geometric invariance. Inspired by the success of Diffusion models on images (Ho)
al.) 2020b;; Song et al. 2020a Song & Ermon 2019), the multi-step generation paradigm has been introduced into generative modeling of atom systems including molecular conformations (Xu
al. 2023). The geometric invariance of the generation path can be guaranteed by designing a
Markov chain with an invariant prior and equivariant transitions (Xu et al., (021). CDVAE (Xie
al., (202)), its CSP adaptation Cond-CDVAE (Luo et al.) 2024a) and SyMat (Luo et al.) 2024b).
generate crystalline materials levearging ((3)-equivariant graph neural network models (Klipfel al. 2023, Gasteiger et al., 2021a) on 3D multi-edge graphs. Utilizing VAE models, they generate lattice parameters, randomly initialize atom coordinates, and iteratively refine these coordinates using score-matching models Song & Ermon 2019). With the fractional coordinate system, DiffCSP (Jiao
et al. 2023) firstly introduces the periodic H(3) equivariance of crystals and designs equivariant diffusion crystal generation model based on periodic diffusion (Jing et al., 2022). Subsequently,
FlowMM (Miller al.) (2024) recently introduced Riemannian Flow Matching (Chen & Lipman| 2023) for the task of crystal generation, offering improved sampling efficiency, albeit at the expense of quality.
We argue that such struggles of balancing between sampling quality and efficiency stems from the lack of proper guidance on each transition from noise prior to data distribution especially for crystals. where thermodynamicall stable materials represent only a small fraction in the search space (Miller al. 2024). For example, early generation states It-1 with low confidence should be preserved less than the later ones to get the next state z;. From the perspective of Bayesian updates, Bayesian Flow Networks (Graves et al. 2023) provides a framework to accurately update each m,-1 according to its confidence parameter a,, the effectiveness of which has been proved in Song al. (2023). However, periodicity is not considered in Graves et al. (2023) and incorporating it into BFN is non-trin non-trivial without desirable distributions with mathematical properties like Gaussian. To address above issues, in this paper, we build a Bayesian flow almost from scratch, identifying and tackling the non-additive accuracy via introducing a novel entropy conditioning mechanism, theoretical reformulations of BFN,
28

---

Published as a conference paper at ICLR 2025
fast sampling algorithm, We demonstrate the effectiveness of the guidance of entropy in Tab. and Tab. and its higher sampling effiency and quality in Figs. [ and 7
Additionally, recently various techniques have also been introduced to boost the performance tailored for crystal generation considering crystal's inductive bias, including Jiao et al. (024);; Cao et al.
(2024); AI4Science et al. (2023) which incorporate the space group constraint into the generation process. Recently, EquiCSP (Lin et al. 2024) proposed to utilize a periodic CoM-free noising method and introduce lattice permutation invariance loss. Those techniques are orthogonal the proposed method in this paper.
Algorithm Training Procedure
1: Require: number of steps n. E R, on E RRt,& 81 E af.,, a E R
2: Input: atom types A, fractional coordinates F, lattice parameter L, length of vocabulary K
n..tt 3: Sample 8 ~ m1,, f +- [i-1]
4: # sampling from Bayesian flow distribution of lattice
5: 7" 41 oi
μu ~ LL (1-1")))
7: # sampling from Bayesian flow distribution of atom type
8: 8 + -3,t²
9: A (B (Kee (KD)),KI)
10: 0² -- -softmaxyyy))
11: # sampling from Bayesian flow distribution of fractional coordinates
12: Y2........
(F af ) (M (F,a
13: m. < atan2(2 a, cos - O3 sin
calculate the accumulated accuracy i.e. entropy
15: - dy y, Ej= 4j sin
16: use network to do inter-dependency modeling across dimensions conditioning on entropy
17: QM +- (HL,, 0". mss mm.c;) c;)
1 (0
(0,",,,,, (0 + (AM
19: calculate the losses of all modalitics
20: of +- Bi (--))
21: YA (a 1KD))aKT)
22: LA < nnnn (y at (KeA 1), af
23:
(1) (1) ; (UA (d) as (Kes - 1), a), (k) -Ex In EExiPo
I1(a)) 24: CF < n Io(a,))
(1 cos(F v (0
25: L1 = (1- 2/n)
| /4-U n/ll
2//
26: Minimize LA + LF + L2
29

---

Published as a conference paper at ICLR 2025
Algorithm/ 2 Sampling Procedure
Require: number of steps é R, length of vocabulary K, E ERt, E ext,af af R
# initialize the prior parameters
Po. Pu
Pa + k 11xN
mo <- (0, co +
c-0
for :ri- <- 1.... do
t <
use network do inter-dependency modeling across dimensions of all modalities
g. + [H:-1, 0 C.-1)
(0.",, V (0,",,, V 10-1) < W(O
ifi < n then
# do Bayesian update for lattice parameter
a oo
-- 28/n
of 2/n)
y (0 1-11, +
1. P1- y
pi-
A: t
Bayesian update for fractional coordinates
y (M (=1),
m. + atan2(af siny +c-1sin m,-1, of cos yos++c-12 cos cos m.)
c, - Il siny + m.-1,; of oos + cos mi-1]/11
do Bayesian update for atom types
a - 31 (2--))
y ~ N (a (Kek 1),
0:11 0' ex e
0, Σ. 0x
o'
end if
A ~ V. 10 # sample from the final probability prediction
Return ( 1, to-1), ( 10 tn-1)
30


Reviews:
[{"review_id": "T58b1yxuau", "reviewer": "Reviewer_VEwm", "summary": "This paper proposes a new crystal generative model for materials using Bayesian Flow Networks, a diffusion-like generative model but supports more types of prior for noise distributions. To effectively enforce E(3) equivariance in generation, the fractional coordinates are generated on the hyper-torus manifold defined in Jing et al. 2022. Instead of standard Gaussian noise used for generation of torsion angles, von Mises distribution is used with derived Bayesian update. To ensure the receiver belief entropy is linearly decreasing, a numerical binary search is done for determining the schedule of the sender’s accuracy (noise) level. In experiments, this method performs better than SOTA methods such as DiffCSP, FlowMM and CDVAE. When compared with diffusion-based method DiffCSP, this method excels at sampling with fewer steps (NFEs).", "questions": "- SE(3) or E(3) equivariance? Is reflection included when generating the fractional coordinates on the hypertorus. Only translation and rotation are mentioned to be preserved in the paper (also in Jing et al. 2022 as well), which corresponds to SE(3).\n- Comparison with baselines with same number of NFEs: how are the methods evaluated, are the stepsizes $\\Delta t$ adjusted by the number of sampling steps as well?\n    \n    Also for small NFEs, it makes more sense to compare with flow matching or diffusion model with ODE sampling, such as FlowMM? Since diffusion models use stochastic sampler which in general requires more steps as seen in the EDM paper[3]. How does FlowMM (which uses ODE sampler) perform with fewer steps?\n    \n- Compare with SOTA: DiffCSP++ [1] should be considered as another baseline to compare to, and also with fewer steps of sampling. Its performance seems to be better than DiffCSP. Also MatterGen-MP should be considered as a baseline for reference.\n- Evaluation on metrics such as uniqueness, novelty and stability?\n- Training time comparison: How does training time compare for different methods? Does the introduced von Mises distribution Bayesian update incur additional training cost per batch?\n\n[1] Jiao, Rui, Wenbing Huang, Yu Liu, Deli Zhao, and Yang Liu. \"Space group constrained crystal generation.\" *arXiv preprint arXiv:2402.03992* (2024).\n\n[2] Zeni, Claudio, Robert Pinsler, Daniel Zügner, Andrew Fowler, Matthew Horton, Xiang Fu, Sasha Shysheya et al. \"Mattergen: a generative model for inorganic materials design.\" *arXiv preprint arXiv:2312.03687* (2023).\n\n[3] Karras, Tero, Miika Aittala, Timo Aila, and Samuli Laine. \"Elucidating the design space of diffusion-based generative models.\" *Advances in neural information processing systems* 35 (2022): 26565-26577.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The major strength of the method is using Bayesian Flow Networks for crystal material generation. The von Mises distribution is used and its Bayesian update is used for training the BFN model, where previous BFN models only use Gaussian distribution. And for this special distribution, the authors identified that it is important to condition the model with the entropy of the receiver’s belief instead of the time. \n\nThe other strength is the improved sampling efficiency of BFN model as compared to diffusion-based method such as DiffCSP. But it is unclear how it compares with ODE sampling method such as flow matching or diffusion with ODE sampler.", "weaknesses": "Major weakness is on experiment evaluation.\n\n- For baselines to compare, the latest DiffCSP++[1] (which gives better performanc) and MatterGen[2] are not included.\n- For metrics, what about uniqueness, novelty, and stability? The goal of material generation is generating novel unique materials that are stable. Hence, these metrics are the most important to measure for use case of material generation.\n- For comparing methods with same number of network forward evaluations, the sampling stepsizes should be adjusted, i.e. $\\Delta t = 0.01$ if sampling $100$ steps. It’s not clear from the current description. Also diffusion models with stochastic samplers are known to need more steps in sampling. For better comparison, maybe consider flow matching (such as FlowMM) or diffusion model with ODE sampler, and with adjusted sampling stepsizes.\n\nThe presentation of BFN can be improved. Maybe with a small toy example to explain how it works. Also for introducing the parameters of von Mises distribution, might be good to refer to Figure 5 and illustrate how each parameter affect the distribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kPpPfnO1u0", "reviewer": "Reviewer_Yjva", "summary": "This paper extends BFNs to von-Mises distributions on the product of 1-spheres (flat torus) and applies this method to modeling conditional material generation (conditioned on atom type, generate positions).", "questions": "About BFN extension:\n  - Can the torus example in Figure 3 be explained through geodesic interpolations of $\\theta_{i-1}$ and $y$?\n  - I don't have sufficient background in BFN to understand the preliminaries, as it immediately uses terminology from BFNs. Can you summarize the basic BFN framework and explain why extending to periodic is not easy in 2-3 sentences in layman terms? \n  - There may be missing information about the sampling in Eq (3). How is y_1 defined? How do you sample the y_i sequence? Is that based on Phi?\n\nExperiments:\n  - Why not also report stability rate for Table 2? As I understand, this is an important metric for material generation.\n  - FlowMM was also proposed for efficient sampling and has a similar (but weaker) plot of Figure 4 (Match rate vs NFE). Can you add FlowMM into Figure 4?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- Method seems novel and using a different model for geodesic generative modeling seems interesting.\n- Results seem good and competitive.", "weaknesses": "- My main concern is I found the definitions and exposition on Bayesian Flow Networks very unclear.\n- Some minor concerns about the experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CXpCrQXiXh", "reviewer": "Reviewer_qhoR", "summary": "This paper introduces a novel generative model for creating crystal structures using a Periodic Bayesian Flow. Its unique contribution is adapting Bayesian Flow Networks (BFNs) to model periodic data on non-Euclidean spaces (specifically, the hyper-torus), which is essential for the spatial symmetry of crystals. Traditional BFNs, optimized for Euclidean data, are insufficient for the periodic nature of crystals, prompting this adaptation to account for non-monotonic entropy and periodicity. The authors introduce majorly these concepts; (a) Periodic Bayesian Flow on a hyper-torus, designed for non-Euclidean spaces to improve generative modeling accuracy for crystal data. (b) Entropy Conditioning instead of time-based conditioning, which better informs the model about the generation state due to non-additive entropy dynamics. (c) Fast Sampling Algorithm and reformulations of BFN for computational efficiency, achieving approximately 100x improvement in speed over previous diffusion-based methods. Experimental results demonstrate CrysBFN's performance advantages on tasks like ab initio crystal generation and crystal structure prediction, consistently outperforming existing methods (e.g., DiffCSP and FlowMM) in accuracy, efficiency, and property statistics across datasets such as Perov-5, Carbon-24, and MP-20. The paper establishes CrysBFN as a state-of-the-art approach in generative crystal modeling, with potential applications for other data types on periodic manifolds. The approach, validated through extensive experimentation, advances both theoretical and practical methodologies for material generation tasks.", "questions": "1. Given the scope of this work I do suggest that the section covering related work may be improved. The authors should do a good survey of the past work in cystal generations particularly in the field of crystal genration with other implicit generative models for crystal representations. Some of which I was able to find by searching for representation based genrative model in citations to CDVAE (Xie et. al.) paper are: 1. https://arxiv.org/abs/2306.04510, 2. https://arxiv.org/abs/2403.10846, 3. https://arxiv.org/abs/2408.07213 (kindly read and search for more). I request the authors to kindly include papers which are in the same field to address the concerns in this paper and how your research aligns or complements with these papers, so that this work becomes complete.\n\n2. The authors are suggested to kindly improve section 2 of their work where they have mentioned some aspects of previous related work, it would be better to include how the work is different and morevoer how does the work takes the research forward from the previous work. If the space becomes an issue (As the authors have already breached the page limitation part, then kindly include a section in complementary section) kindly add a complementary section.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall, this work is thorough, well-written, and stands out as a valuable contribution to the literature on generative material models. The paper’s strengths lie in its originality, quality, clarity, and significance. Its originality is evident in the introduction of CrysBFN—the first periodic Bayesian Flow Network designed for modeling non-Euclidean crystal data on a hyper-torus. The following points highlight the paper’s most impressive strengths.\n\n1. The paper introduces CrysBFN, the first Bayesian Flow Network designed for periodic, non-Euclidean crystal data.\n2. Authors propose a novel entropy-based conditioning that enhances modeling accuracy for periodic structures.\n3. Achieves a 100x speedup in sampling efficiency compared to previous diffusion-based methods.\n5. State-of-the-Art Results, outperforms leading models in crystal generation accuracy and structural validity across multiple datasets.", "weaknesses": "1. Although not in the scope of the paper, but the paper may discuss on how CrysBFN could generalize to other non-crystal periodic or symmetrical data types. While CrysBFN shows strong results for specific crystal datasets, the paper could benefit from discussing its potential for generalization to other periodic or non-Euclidean data types beyond crystals. Including experiments or examples of how CrysBFN could extend to other symmetrical structures, such as molecular or lattice-based materials, would add depth and show broader applicability.\n\n2. The paper introduces entropy conditioning in place of traditional time-based conditioning, which adds complexity to the model. While the necessity of entropy conditioning is discussed, providing additional comparative analysis between the two methods across varied datasets (e.g., simpler vs. more complex structures) could clarify its practical advantages and help practitioners understand when to apply entropy conditioning.\n\n3. Though the model achieves a high sampling efficiency, the paper does not provide a detailed analysis of the overall computational cost for training and deployment. Adding a breakdown of the computational resources required, such as training time or GPU hours, would provide a clearer picture of the model’s practicality for large-scale or industrial applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)