Title: Unleashing the power of novel conditional generative approaches for new materials discovery
Venue: NeurIPS 2024
Accepted: no
Keywords: new materials design

new materials

solid state physics

generative ai

crystal structure generation

diffusion

flow matching
Abstract: For a very long time, computational approaches to the design of new materials have relied on an iterative process of finding a candidate material and modeling its properties. AI has played a crucial role in this regard, helping to accelerate the discovery and optimization of crystal properties and structures through advanced computational methodologies and data-driven approaches. To address the problem of new materials design and fasten the process of new materials search, we have applied latest generative approaches to the problem of crystal structure design, trying to solve the inverse problem: by given properties generate a structure that satisfies them without utilizing supercomputer powers. In our work we propose two approaches: 1) conditional structure modification: optimization of the stability of an arbitrary atomic configuration, using the energy difference between the most energetically favorable structure and all its less stable polymorphs and 2) conditional structure generation. We used a representation for materials that includes the following information: lattice, atom coordinates, atom types, chemical features, space group and formation energy of the structure. The loss function was optimized to take into account the periodic boundary conditions of crystal structures. We have applied Diffusion models approach, Flow matching, usual Auto-Encoder (AE) and compared the results of the models and approaches. As a metric for the study, physical pymatgen matcher was employed: we compare target structure with generated one using default tolerances. So far, our modifier and generator produce structures with needed properties with accuracy 41% and 82% respectively. To prove the offered methodology efficiency, inference have been carried out, resulting in several potentially new structures with formation energy below the AFLOWLib-derived convex hulls.

Abstract

For a very long time, computational approaches to the design of new materials have relied on an iterative process of finding a candidate material and modeling its properties. AI has played a crucial role in this regard, helping to accelerate the discovery and optimization of crystal properties and structures through advanced computational methodologies and datadriven approaches. To address the problem of new materials design and fasten the process of new materials search, we have applied latest generative approaches to the problem of crystal structure design, trying to solve the inverse problem: by given properties generate a structure that satisfies them without utilizing supercomputer powers. In our work we propose two approaches: 1) conditional structure modification: optimization of the stability of an arbitrary atomic configuration, using the energy difference between the most energetically favorable structure and all its less stable polymorphs and 2) conditional structure generation. We used a representation for materials that includes the following information: lattice, atom coordinates, atom types, chemical features, space group and formation energy of the structure. The loss function was optimized to take into account the periodic boundary conditions of crystal structures. We have applied Diffusion models approach, Flow matching, usual Autoencoder (AE) and compared the results of the models and approaches. As a metric for the study, physical pymatgen matcher was employed: we compare target structure with generated one using default tolerances. So far, our modifier and generator produce structures with needed properties with accuracy 41% and 82% respectively. To prove the offered methodology efficiency, inference have been carried out, resulting in several potentially new structures with formation energy below the AFLOW-derived convex hulls.

1 Introduction

The search for novel materials with specified properties has been a cornerstone of scientific exploration for decades. From the discovery of semiconductors revolutionizing electronics to the development of superalloys enhancing aerospace technologies, the synthesis of new materials has continually propelled technological advancements.

However, traditional methods for material discovery often employ exhaustive trial and error experimental approaches. In turn, computational efforts, relying on density functional theory (DFT)[1] approaches, usually require huge amounts of computing power. In this regard, automatic descriptor generators[2], GNNs[3][4] and transferable GNN models [5] fueled combination of these methods and machine learning (ML) approaches. In particular, the

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

utilization of generative machine learning models, such as Variational Autoencoder[6] and GANs[7], presents a paradigm shift in how crystal structures are generated and optimized. By harnessing the power of data-driven approaches, we can navigate the vast landscape of possible crystal structures with unprecedented efficiency and precision.

Recent advancements in the field of materials discovery have yielded promising results through various innovative approaches. For instance, FTCP[6] utilizes Autoencoders for uncovering new materials, while CubicGAN[7] leverages GANs for the discovery of cubic crystal materials. Additionally, Physics Guided Crystal Generative Model (PGCGM)[8] has introduced a method for generating crystal structures based on specific space groups encoding. DP-CDVAE[9] is a model, that combines VAE and diffusion approaches. MatterGen[10] employed equivariant GNNs as score matching function in diffusion processes for crystal structure generation.

One of the most discussed frameworks is GNoME[11] that has made most recent and large advancements in the field of the new materials discovery employs a sophisticated pipeline to discover new materials, particularly focusing on inorganic crystals. This allows for the discovery of innovative materials beyond known structures.

After generating candidate structures through both pipelines, GNoME evaluates their stability by predicting their formation energies. Based on the comparison of the obtained formation energy with those of the known competing phases (i.e. stability assessment), the model selects the most promising candidates for further evaluation using known theoretical frameworks.

The question of the completeness of chemical space arises due to two main concerns with GNoME-derived stable structures. Firstly, they mostly contain three or more unique elements, while ternary and quaternary structures are less explored than binary compounds. Secondly, the comparison of GNoME-discovered structures to the Materials Project, which has 154,718 materials, is flawed since larger databases like AFLOW, NOMAD, and the Open Quantum Materials Database contain millions of entries. This raises questions about the novelty of the discovered materials.

In this study, we present an end-to-end framework for the generation of crystal structures with specified properties using advanced generative AI techniques. The basis architecture of the models is Autoencoder, enabling encoding and decoding structural representations. Then, the most commonly used generative approaches in image generation were utilized to model probability distribution transformations, and to capture complex underlying structure-property relationships within our dataset: Flow Matching[12], Denoising Diffusion Probabilistic Models(DDPM)[13], and Denoising Diffusion Implicit Models(DDIM)[14]. Through the integration of these techniques, we aim to transcend conventional limitations in materials discovery, paving the way for accelerated predictions of materials with desired properties.

To employ model architectures often used for image/video generation, a matrix representation of crystal structures was developed, containing crucial information such as chemical composition, atomic coordinates, symmetries (space group), and formation energies. Within the approach proposed, it has become important to develop a novel metric for assessing the similarity between generated structures and target configurations. This metric obviates the need for computationally expensive DFT calculations, allowing for rapid validation and refinement of generated structures. Furthermore, we introduce a loss function that accounts for the periodic boundary conditions inherent in crystal lattices, ensuring the fidelity of the generated structures.

Our study explores two distinct approaches for crystal structure prediction: 1) conditional structure modification and 2) conditional structure generation. The former involves optimizing the stability of existing structures by generating more stable polymorphs, while the latter entails the generation of entirely new structures based on user-defined criteria. Through rigorous analysis, we demonstrate the efficacy of our approach in discovering novel materials with desired properties.

Importantly, to validate the utility of our framework, we conducted a series of generation experiments using the Vienna Ab initio simulation package(VASP)[15] as a tool for inference validation. Remarkably, our method facilitated the discovery of 6 structures below the corresponding convex hull. This significant outcome underscores the remarkable potential of

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

our framework in uncovering thermodynamically stable materials, thereby offering promising avenues for advanced materials discovery and design.

2 Data, Dataset

2.1 Data overview

In this study, the AFLOW database[16] was utilized as a source of data on the structures and properties of materials. AFLOW is an extensive and comprehensive database that consolidates a vast array of materials-related information, offering an expansive repository for crystallographic data, computed properties, and various other materials-science-related datasets. AFLOW database contains more than 3.5 million structures.

From the extensive collection housed within AFLOW, the focus was narrowed to select only polymorphs, because models are trained to distinguish composition-property and structureproperty relations with numerous structures of the same chemical composition. Specifically, the selection process targeted polymorphic structures with 4 to 60 atoms within their unit cells. This criterion aimed to encompass a diverse yet manageable subset of structures, balancing complexity with computational feasibility. By filtering polymorphs based on their atom count, the dataset was balanced.

Moreover, in order to decrease the complexity of the data, we have removed all structures containing elements and space groups found in less than 1% of all structures. The entire dataset consisted of more than 85000 polymorph groups including more than 2.1 million structures. The minimum size of group of polymorphs was 7 samples and the maximum one was 71 samples. The total number of space groups was 19 and the total number of chemical species over the dataset was 55. Each structure S in the dataset is described by the following features:

Fractional coordinates of atoms in the lattice basis X coord (has 60 rows with 3 coordinates x, y, z each) and X lattice (matrix 3 by 3 constructed of 3 base vectors). Overall matrix X of structure is constructed as

Chemical elements which are presented as a one-hot matrix elements ij of size 64 × 103 (including padding), where ones are positioned at the indices corresponding to the position of a certain chemical element in the periodic table.

elements ij = { 1 if i -th atom's element number from the periodic table = j 0 otherwise

Elemental property matrix elementalProperties containing 22 chemical features encoding chemical elements obtained from [8]. The properties of each element were calculated using Mendeleev package[17].

Space group spg of a structure. We use the space group encoding method presented in [8], when each space group is represented by a 192 × 4 × 4 matrix, which corresponds to 192 possible symmetry operations.

Structure formation energy E

Nsites - number of atoms in a crystal lattice.

2.2 Data representation. Modification task

The crystal pair sampling strategy involves handling a potential data leakage: possible inclusion of structures from the same polymorph group but with different energies into training and validation subsets. To mitigate this issue, the polymorph group formulas were initially divided into distinct training and validation sets, ensuring a relatively balanced distribution of chemical elements across these subsets. Subsequently, the pairs were categorized into two groups: those with low-energy (lowest energy in polymorph group) targets designated as

lowestEnergyPairs = ( S i , S 0 ) ∀ i ∈ [1 , ..., structuresNum ] and those with non-low-energy 133 targets, all structures except the most optimal one, formed as nonLowestEnergyPairs = 134 ( S i , S j ) | i > j > 0 . The validation set was constructed as a subset of lowestEnergyPairs . 135 The training set was dynamically constructed every epoch from lowestEnergyPairs and 136 nonlowestEnergyPairs , preserving equal numbers of pairs sampled and maintaining a 137 limited count per polymorph group. This strategy ensured a robust separation between 138 training and validation sets, thus preventing data leakage and improving model performance. 139

140

141

142

Each pair sample { S init , S target } ∈ pairDataset consisted of the information about each structure (hereinafter, we will call them initial and target structures). The following data was used:

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

Coordinates and lattice information of initial and target structures X init , X target

Difference in formation energies between initial and target structures E diff = E target -E init

Space group of target structure spg target

Elements matrix elemetsMatrix , elemental property matrix elementalProperties and number of sites numSites , which are the same for initial and target structure because of identical chemical composition.

The modification task involved transforming the input structure X init into the target structure X target .

2.3 Data representation. Generation task

In its tern the generation task receives normal or uniform (depends on a model) noise as input from which the structure is generated, which is akin to the image generation processes in computer vision tasks.

For the generation task, an additional dataset was constructed. Data for the generation task is slightly simpler, while it considers only { S target } . Therefore, the models can be trained on all structures available, rather than pairs. The following data is used:

Coordinates and lattice information of target structure X target

Formation energy of target structure E target

Space group of target structure spg target

Elements matrix elemetsMatrix , elemental property matrix elementalProperties and number of sites numSites of target structure.

3 Loss and metrics

3.1 Atomic coordinates

The atomic coordinates are represented as a 60 × 3 matrix, where each row corresponds to the coordinates of an atom. The L1 loss was utilized during the training of a model for predicting atomic coordinates.

L 1 ( preds, target ) i = || preds i -target i || 1 = ∑ 3 j =1 | preds ij -target ij | , where target and pred are target and predicted atomic coordinate matrices.

3.2 Lattice

The lattice itself is represented as a 3x3 matrix, where each row signifies a directing basis vector. In this case, we have also used the L1 norm as a loss function.

3.3 Periodic boundary condition loss

This section presents an enhanced loss function, designed for the regression model (see 175 Section 5.1), that addresses this challenge by integrating periodic boundary conditions into 176

the loss calculation, outperforming the conventional L1 loss function. In the field of ML 177 applied to atomic structures, even slight displacement of atomic coordinates is crucial and 178 employing appropriate loss functions that consider the periodic nature of atomic structures 179 increases the flexibility of model predictions. 180

181

182

183

184

185

186

187

In the dataset representing atomic structures, it is crucial to acknowledge the presence of atoms residing at various positions within the lattice framework. Certain atoms are positioned at the vertices, edges, or faces of the lattice. According to periodic boundary conditions (PBC), identical atoms in the vicinity of vertices, edges, or faces but also exist in analogous positions across the lattice. Implementation of such an invariance within the loss function helps in effectively capturing periodic pattern of crystals, enhancing the model's capability to learn and predict atomic structures more comprehensively.

(a)

(b)

(c)

=

The loss function is being calculated as minimum of distances from predicted point to the 188 target one taking into account 26 its periodic images (according to PBC) A.4. 189

190

191

192

The empirical validation of this enhanced loss function showcases its superiority(Figure4) in capturing discrepancies within atomic structures, thus indicating its potential as a robust tool for improving the accuracy of ML models in materials science applications.

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

3.4 Metric

As a metric, we have chosen an analogue of accuracy: the generated structures are compared to the target structures using a specialized matcher, yielding the proportion of structures that successfully pass the matching process. For metric calculation, we employed the Pymatgen StructureMatcher with the default set of parameters ( ltol = 0 . 2 , stol = 0 . 3 , angle _ tol = 5 ). Although this approach is less accurate than structure relaxation using ab initio calculations and comparing the structure formation energy with the energy above the hull, it enables model validation to be performed orders of magnitude faster than the traditional method.

4 Model

For experiments, a 1d UNet model (see Figure2 (b)) architecture similar to the 2d UNet model described in [18] was utilized along with 2D and 1D convolutional neural networks (CNNs) for the space group and element matrix embeddings, respectively. Based on this model, 3 different training processes have been developed: ordinary regression model, Conditional Flow Matching (CFM)[19] model, and diffusion model.

The model was conditioned (see Figure2 (a)) on the following data: time condition ( t ), the same as in [18], element condition ( el ), formation energy difference condition ( E diff ), and desirable space group ( spg ). el emb , spg emb and E diff are concatenated into one embedding

C emb . t is fed into the Transformer Positional Encoding Layer and transformed into an 210 embedding T emb . The two embeddings: C emb and T emb are then applied into one condition. 211

(a) Condition block

(b) UNet

Energy

Diff /

Energy

SPG

2DConv

Concatenate

Condition

Elements

1DConv

Positional

Encoding

coordinates

and lattice

Condition

十

Down

Block

Up Block

Condition

+

Down

Block

UpBlock

Condition

+

Down

Block

UpBlock

Condition

Down

Up Block

Block

5 Methodology

In this work, two approaches are proposed: crystal structure generation and crystal structure modification. For the generation approach, crystal structures are generated from normal or uniform noise and conditioned to t , el , E , spg . Within the generation, we employed three algorithms: DDPM, DDIM, and CFM models. For the modification approach, crystal structures are generated by modifying other structures, while conditioning to el , E diff , spg (and optionally t , not used in ordinary regression UNet). For the modification task, we have employed three algorithms: UNet Regression model, diffusion model, based on Palette[20] approach, and CFM model. For the generation task, we have employed four algorithms: diffusion models with DDPM and DDIM samplers, and CFM models on Uniform and Normal noise.

5.1 Regression model

During the training stage, the structure coordinates and lattice x 0 , elements features el , space group spg and E diff are used as conditions. The model is trained to return x 1 structure coordinates and lattice (Algorithm 1). As for the inference process, one can see the details in the Algorithm 2

5.2 Conditional Flow Matching models

CFM is a fast method for training Continuous Normalizing Flows (CNF)[21] models without the need for simulations. It offers a training objective that enables conditional generative modeling and accelerates both training and inference.

The basic way of training CFM model (Algorithm 3) organized as follows: during the training stage, x 0 and x 1 are sampled from the source distribution and the target distribution respectively, then a linear interpolation x t is calculated as x t = tx 1 +(1 -t ) x 0 (exponential moving average between distributions x 0 and x 1 ; t is sampled from a uniform distribution U (0 , 1) ), and afterwards pass the x t and t as inputs to our model f θ , forcing the model to predict a velocity from the distribution x 0 to x 1 . Therefore, the loss for CFM model is the following: L CFM = E t,x 1 ,x 0 [ || f θ ( x t , t ) -( x 1 -x 0 ) || 2 ] = E t,x 1 ,x 0 [ || f θ ( tx 1 +(1 -t ) x 0 , t ) -( x 1 -x 0 ) || 2 ]

For the modification approach, x 0 and x 1 are both sampled from our dataset distribution according to the sampling strategy for modification mentioned in 2.2. Also, the model is conditioned to el, spg 1 , E diff , besides t (see Algorithm 4)

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

For the generation approach, we tested two noise distributions for the x 0 : normal distribu243 tion N (0 , 1) and uniform noise distribution U (0 , 1) , which resulted in significantly better 244 performance. The intuition for using uniform distribution instead of normal one was inspired 245 by the diagram of x, y, z coordinate distribution (Figure 3). The model is also conditioned 246 to el, spg 1 , E , and t (see Algorithm 5) 247

248

249

250

251

During the sampling stage, we generate X 1 structure by the given X 0 by solving the following ordinary differential equation (ODE): dx t = f θ ( x t , t, el, spg 1 , E ) dt , beginning with x 0 . In order to solve the ODE, the Euler method was employed: x t + h = x t + hf θ ( x t , t, el, spg 1 , E ) (Algorithm 6)

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

5.3 Diffusion models

In our work, we observe diffusion models. Diffusion models generate samples from a target distribution x 1 , starting from a source distribution x 0 ∼ N (0 , I ) .

During training, these models are trained to reverse a Markovian forward process, which adds noise x 0 to the data step by step. Meaning, diffusion models are trained to predict the noise added to the data samples x 1 . In order to train a model in this setup, the following loss function is used, L simple = E t,x 1 ,x 0 [ || x 0 -f θ ( √ ¯ α t x 1 + √ 1 -¯ α t x 0 , t ) || 2 ] where ¯ α t = ∏ t s =1 α s and α t = 1 -β t ( β t is the variance by which added noise is being scheduled on each step t ).

Our modification approach is based on Palette, which enables sample-to-sample generation (from noise ϵ ∼ N (0 , 1) ) using x 0 structure coordinates and lattice, el , spg 1 , E diff and t as conditions for generation of x 1 using the DDPM algorithm. Sampling stage is performed by a backward diffusion process with linear scheduler (see Algorithms 7, 8).

For the generation approach ((Algorithm 9), x 0 is sampled from a normal distribution and el , spg 1 , E , t are fed into the model as conditions. During our experiments, we tested 2 approaches: DDPM(Algorithm 10) classic approach and DDIM(Algorithm 11) which results in usage of smaller number of sampling steps in order to speed up the generation process. Moreover, DDIM enables the process of generating samples from random noise to be deterministic.

6 Experiment Results

All the models presented in tables (Table 1 and Table 2) have been trained with the same 271 hyperparameters and architectures. The metric used is described in Section 3.4. We also 272 provide all experiment details in A.3. 273

274

7 Inference

In order to demonstrate a potential of the proposed approaches, we have chosen a chemical 275 composition, containing numerous variations and phases of structures composed of [W, B, 276 Ta] with well-explored convex hull. Structures that lie on the convex hull are considered to 277 be thermodynamically stable, and the ones above it are either metastable or unstable. 278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

7.1 Inference pipeline

The proposed testing procedure involves generating test conditions for structures, passing them to the trained generative models, pre-optimizing the generated structures to accelerate the following ab initio calculations, and final relaxation and formation energy calculating using VASP. Although in this work two approaches were proposed: Generation and Modification, the following pipeline has only been applied to generation models, due to the fact, that modification approach is based on structure-polymorphs, which leads to the necessity to have at least one structure with needed composition, which is not always so. That fact makes generation models much more flexible in generation structures not only with needed properties, but also with needed composition. Another advantage of the generation models is value of metric that is two times bigger than in modification tasks. The inference algorithm is as follows:

1. Test Condition Formation:

The chosen chemical formulas were utilized for feature extraction of el . Three chemical compositions have been used: 1) Ta 1 W 1 B 6 , 2) Ta 1 W 2 B 5 and 3) Ta 2 W 1 B 5 .

We have taken spg presented in the dataset as an additional condition, obtaining 19 space groups.

Finally, a set of target formation energies E has been formed. We have carried out three experiments: 1) starting from the energy of the convex hull and decreasing with a step of 0.01 eV/atom, 2) starting from the energy of the convex hull and decreasing with a step of 0.1 eV/atom, and 3) starting from the energy 1 eV/atom less than the energy of the convex hull and decreasing with a step of 0.01 eV/atom. In total, 21 energy values were used for every inference run.

Final inference conditions were obtained by making all possible combinations of spg and E for a certain composition el

Model Inference: The conditions from the step 1 have been put to one of the trained models, resulting in the generation of structures. Two models have been employed: Diffusion approach and Flow matching

Pre-Optimization: Following the generation of all structures, each structure has been pre-optimized using the PyMatGen structure relaxation method. The method used m3gnet [22] model with default parameters. PyMatGen pre-optimization contributed to overall speedup of further VASP relaxation.

Structure relaxation: Pre-optimized structures were relaxed using VASP (the recommended pseudopotentials, plane wave energy cutoff of 500 eV, Ediif and Ediffg convergence criteria of 10 -5 and -10 -2 were used).

7.2 Inference results

To summarize, 6 experiments have been carried out for two different models and for three formation energy conditionings. Every experiment includes 3*380 structures, per 380 structures for every single chemical composition. The results of experiments can be seen in Table 3

As can be seen, 4 structures were obtained with formation energies significantly lower than those obtained from the AFLOW-derived convex hull. Thus, it can be concluded that this observation indicates the potential stability of the generated structures rather than differences in the computational methods used in this work and during AFLOW generation. Another four structures also have energies below the convex hull, but in the vicinity of it. Thus, their potential stability should be interpreted with caution.

8 Data availability

The raw crystal dataset is downloaded from https://aflowlib.org

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

9 Code availability

The source code for training and inferencing our models can be obtained from GitHub at https://github.com/AIRI-Institute/conditional-crystal-generation

10 Conclusion

In this article, we have offered two approaches to generate crystal structures: conditional generation and conditional modification. The first approach is significantly more flexible as it does not require structure-polymorphs, enabling the generation of structures without restrictions on chemical composition, which can be crucial in certain scenarios. Another advantage of the first approach is the simplicity of data preprocessing; it only requires the chemical composition, space group, atom coordinates, and formation energies.

Our methodology has experimentally proven its effectiveness, resulting in four confident potentially new crystal structures with the following energies above the hull: {-1.409, -5.497, -5.426, and -4.852} meV/atom, and four uncertain candidates with energies of {-0.483, -0.466, -0.387, and -0.042} meV/atom. We have demonstrated that conditional generation approaches, commonly used in image generation, are also fruitful in the design of new materials.

Although the proposed methodology demonstrates its efficiency in generating potentially new crystal structures, it has certain limitations. Firstly, the data is represented in a matrix form, which does not account for all possible symmetries of the crystal structures. Secondly, the structures in the dataset range from 4 to 60 atoms per unit cell, with most structures containing fewer than 8 atoms per unit cell. However, to perform well on structures with a large number of atoms per unit cell, the models just should be pretrained on a dataset that includes larger structures.

Furthermore, despite the limited number of experiments(6) and structures generated (7182), 352 we succeeded in identifying hypothetically new structures. We hope that our article will 353 help to reveal the potential of generative AI in design of new materials with targeted 354 thermodynamic properties and inspire other researchers to be part of this innovative journey 355 in materials design. We believe that rapid and efficient generation of novel materials can lead 356 to breakthroughs in various fields such as electronics, pharmaceuticals, and energy storage. 357 This can accelerate technological advancements and make cutting-edge technologies more 358 accessible and affordable. 359

360

References

Diola Bagayoko. Understanding density functional theory (dft) and completing it in 361 practice. AIP Advances, 4(12), 2014. 362

363

364

365

Alexander Dunn, Qi Wang, Alex Ganose, Daniel Dopp, and Anubhav Jain. Benchmarking materials property prediction methods: the matbench test set and automatminer reference algorithm. npj Computational Materials, 6(1):138, 2020.

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

Roman A Eremin, Innokentiy S Humonen, Alexey A Kazakov, Vladimir D Lazarev, Anatoly P Pushkarev, and Semen A Budennyy. Graph neural networks for predicting structural stability of cd-and zn-doped γ -cspbi3. Computational Materials Science, 232:112672, 2024.

Alexey N Korovin, Innokentiy S Humonen, Artem I Samtsevich, Roman A Eremin, AI Vasilev, Vladimir D Lazarev, and Semen A Budennyy. Boosting heterogeneous catalyst discovery by structurally constrained deep learning models. Materials Today Chemistry, 30:101541, 2023.

Alexandre Duval, Simon V Mathis, Chaitanya K Joshi, Victor Schmidt, Santiago Miret, Fragkiskos D Malliaros, Taco Cohen, Pietro Li` o, Yoshua Bengio, and Michael Bronstein. A hitchhiker's guide to geometric gnns for 3d atomic systems. arXiv preprint arXiv:2312.07511, 2023.

Zekun Ren, Siyu Isaac Parker Tian, Juhwan Noh, Felipe Oviedo, Guangzong Xing, Jiali Li, Qiaohao Liang, Ruiming Zhu, Armin G Aberle, Shijing Sun, et al. An invertible crystallographic representation for general inverse design of inorganic crystals with targeted properties. Matter, 5(1):314-335, 2022.

Yong Zhao, Mohammed Al-Fahdi, Ming Hu, Edirisuriya MD Siriwardane, Yuqi Song, Alireza Nasiri, and Jianjun Hu. High-throughput discovery of novel cubic crystal materials using deep generative neural networks. Advanced Science, 8(20):2100566, 2021.

Yong Zhao, Edirisuriya M Dilanga Siriwardane, Zhenyao Wu, Nihang Fu, Mohammed Al-Fahdi, Ming Hu, and Jianjun Hu. Physics guided deep learning for generative design of crystal materials with symmetry constraints. npj Computational Materials, 9(1):38, 2023.

Teerachote Pakornchote, Natthaphon Choomphon-Anomakhun, Sorrjit Arrerut, Chayanon Atthapak, Sakarn Khamkaeo, Thiparat Chotibut, and Thiti Bovornratanaraks. Diffusion probabilistic models enhance variational autoencoder for crystal structure generative modeling. Scientific Reports, 14(1):1275, 2024.

Claudio Zeni, Robert Pinsler, Daniel Z¨ ugner, Andrew Fowler, Matthew Horton, Xiang Fu, Sasha Shysheya, Jonathan Crabb´ e, Lixin Sun, Jake Smith, et al. Mattergen: a generative model for inorganic materials design. arXiv preprint arXiv:2312.03687, 2023.

Amil Merchant, Simon Batzner, Samuel S Schoenholz, Muratahan Aykol, Gowoon Cheon, and Ekin Dogus Cubuk. Scaling deep learning for materials discovery. Nature, 624(7990):80-85, 2023.

Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow 400 matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022. 401

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In 402 H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in 403 Neural Information Processing Systems, volume 33, pages 6840-6851. Curran Associates, 404 Inc., 2020. 405

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. 406 In International Conference on Learning Representations, 2021. 407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

Guangyu Sun, Jen¨ o K¨ urti, P´ eter Rajczy, Miklos Kertesz, J¨ urgen Hafner, and Georg Kresse. Performance of the vienna ab initio simulation package (vasp) in chemical applications. Journal of Molecular Structure: THEOCHEM, 624(1-3):37-45, 2003.

Stefano Curtarolo, Wahyu Setyawan, Gus LW Hart, Michal Jahnatek, Roman V Chepulskii, Richard H Taylor, Shidong Wang, Junkai Xue, Kesong Yang, Ohad Levy, et al. Aflow: An automatic framework for high-throughput materials discovery. Computational Materials Science, 58:218-226, 2012.

Ł ukasz Mentel. mendeleev - a python resource for properties of chemical elements, ions and isotopes.

Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pages 8162-8171. PMLR, 2021.

Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations, 2023.

Chitwan Saharia, William Chan, Huiwen Chang, Chris A. Lee, Jonathan Ho, Tim Salimans, David J. Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models, 2022.

Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.

Chi Chen and Shyue Ong. A universal graph deep learning interatomic potential for the periodic table. Nature Computational Science, 2:718-728, 11 2022.

A Appendix section 1 432

A.1 Distribution of atomic coordinates 433

106

Number of positions

105

104

103

102

Zcoordinate

101

Ycoordinate

Xcoordinate

100

0.0

0.2

0.4

0.6

0.8

1.0

fractional coordinate

A.2 Pseudocode 434

Algorithm 1 Training Regression Modification Model

1: repeat

2: x 0 ∼ q ( x 0 ); x 1 ∼ q ( x 1 ); el ∼ q ( el ); spg 1 ∼ q ( spg 1 ); E ∼ q ( E )

3: L ← || x 1 -f θ ( x 0 , t, el, spg 1 , E ) ||

4: θ ← Update ( θ, ∇ θ L ( θ ))

5: until converge

Algorithm 2 Inferencing Regression Modification Model

1: x 0 ∼ q ( x 0 ); el ∼ q ( el ); spg 1 ∼ q ( spg 1 ); E ∼ q ( E )

2: x 1 = f θ ( x 0 , t, el, spg 1 , E )

3: return x 1

Algorithm 3 CFM Training

1: repeat

2: x 0 ∼ q ( x 0 ); x 1 ∼ q ( x 1 )

3: t ∼ U (0 , 1)

4: x t = tx 1 +(1 -t ) x 0

5: L CFM ←|| f θ ( x t , t ) -( x 1 -x 0 ) ||

6: θ ← Update ( θ, ∇ θ L CFM ( θ ))

7: until converge

Algorithm 4 Training CFM for Modification

1: repeat

2: x 0 ∼ q ( x 0 ); x 1 ∼ q ( x 1 ); el ∼ q ( el ); spg 1 ∼ q ( spg 1 ); E ∼ q ( E )

3: t ∼ U (0 , 1)

4: x t = tx 1 +(1 -t ) x 0

5: L CFM ←|| f θ ( x t , t, el, spg 1 , E ) -( x 1 -x 0 ) ||

6: θ ← Update ( θ, ∇ θ L CFM ( θ ))

7: until converge

Algorithm 5 Training CFM for Generation

1: repeat

2: x 0 ∼ N (0 , 1) or x 0 ∼ U (0 , 1)

3: x 1 ∼ q ( x 1 ); el ∼ q ( el ); spg 1 ∼ q ( spg 1 ); E ∼ q ( E )

4: t ∼ U (0 , 1)

5: x t = tx 1 +(1 -t ) x 0

6: L CFM ←|| f θ ( x t , t, el, spg 1 , E ) -( x 1 -x 0 ) ||

7: θ ← Update ( θ, ∇ θ L CFM ( θ ))

8: until converge

Algorithm 6 Sampling with CFM for Modification or Generation

1: h = 1 T

2: x 0 ∼ q ( x 0 ) or x 0 ∼ N (0 , 1) or x 0 ∼ U (0 , 1)

3: el ∼ q ( el ); spg 1 ∼ q ( spg 1 ); E ∼ q ( E )

4: for do t = 1 , . . . , T do

5: x t +1 = x t + hf θ ( x t , t, el, spg 1 , E )

6: end for

7: return x 1

Algorithm 7 Training DM for Modification

1: repeat

2: x 0 ∼ q ( x 0 ); x 1 ∼ q ( x 1 ); el ∼ q ( el ); spg 1 ∼ q ( spg 1 ); E ∼ q ( E )

3: t ∼ U ( { 1 , . . . , T } )

4: ϵ ∼ N (0 , I )

5: L D ←|| ϵ -f θ ( √ ¯ α t x 1 + √ 1 -¯ α t ϵ, x 0 , t, el, spg 1 , E ) ||

6: θ ← Update ( θ, ∇ θ L D ( θ ))

7: until converge

Algorithm 8 Sampling with DM for Modification

1: x T ∼ N (0 , I )

2: for do t = T, . . . 1 do

4: z ∼ N (0 , I ) if t > 1 else z = 0

6: end for

7: return x 1

Algorithm 9 Training DM for Generation

1: repeat

2: x 1 ∼ q ( x 1 ); el ∼ q ( el ); spg 1 ∼ q ( spg 1 ); E ∼ q ( E )

3: t ∼ U ( { 1 , . . . , T } )

4: ϵ ∼ N (0 , I )

5: L D ←|| ϵ -f θ ( √ ¯ α t x 1 + √ 1 -¯ α t ϵ, t, el, spg 1 , E ) ||

6: θ ← Update ( θ, ∇ θ L D ( θ ))

7: until converge

Algorithm 10 DDPM Sampling

1: x T ∼ N (0 , I )

2: for do t = T, . . . 1 do

3: x 1 ∼ q ( x 1 ); el ∼ q ( el ); spg 1 ∼ q ( spg 1 ); E ∼ q ( E )

6: end for

7: return x 1

Algorithm 11 DDIM Sampling

A.3 Experiment Details 435

All the experiments use the same hyperparameters for the model: 436 · num_res_blocks = 7 437 · attention_resolution = (1, 2, 4, 8) 438 · model_channels = 128 439 In all the experiments models are trained with the same training parameters: 440 · optimizer = Adam 441 -betas = (0.9, 0.999) 442 -eps = 1e-08 443 -weight_decay = 0 444 · batch_size = 256 445 · epochs = 400 446 · learning_rate = 1e-4 447 · lr_warmup_steps = 500 448 · random_state = 42 449 An important note, that all our experiments have been conducted in mixed precision in fp16. 450 Generation task: Diffusion Model (DDPM): 451 · num_train_timesteps = 1000 (diffusion process discretization) 452 · beta_start = 0.0001 453 · beta_end = 0.02 454 · num_inference_steps = 100 455 · beta_schedule = "squaredcos_cap_v2" (cosine) 456 Diffusion Model (DDIM): 457 · num_train_timesteps = 1000 (diffusion process discretization) 458 · beta_start = 0.0001 459 · beta_end = 0.02 460 · num_inference_steps = 100 461 · beta_schedule = "squaredcos_cap_v2" (cosine) 462 Flow Matching x 0 ∼ N (0 , 1) : 463 · num_inference_steps = 100 464

Flow Matching x 0 ∼ U (0 , 1) : 465

num_inference_steps = 100 466 Modification task: 467 Regression UNet: 468 · num_inference_steps = 1 469 Diffusion Model: 470 · num_train_timesteps = 1000 (diffusion process discretization) 471 · beta_start = 0.0001 472 · beta_end = 0.02 473 · num_inference_steps = 100 474 · beta_schedule = "squaredcos_cap_v2" (cosine) 475 Flow Matching: 476 · num_inference_steps = 100 477 A.4 PBC Loss details 478

The PBC loss function operates through several steps: 479

Vertices evaluation: If the target coordinate of the atom is lattice vertex (all 3 coordinates x, y, z are equal to 1 or 0), then loss between prediction point preds i and target point target i is being calculated using following formula:

where vertices is a set of 8 possible positions according to PBC ( { 0 , 0 , 0 } , { 0 , 0 , 1 } , ..., { 1 , 1 , 1 } ).

Edges evaluation: If the target coordinate of the atom is located on lattice edge (two coordinates are equal to 1 or 0 and one is not). For example, a lattice edge atom at point { 0 , 1 , 0 . 3 } has identical atoms at points { 0 , 0 , 0 . 3 } , { 1 , 0 , 0 . 3 } , { 1 , 1 , 0 . 3 } . As we can see, in this example z -coordinate is fixed but x and y are exchangeable. Therefore, if the target point is represented as { x, y, z } , we can use the following formula:

where edgePoints is a set of 4 possible positions according to PBC.

Case of fixed point x : edgePoints = {{ x, 0 , 0 } , { x, 0 , 1 } , { x, 1 , 0 } , { x, 1 , 1 }}

Case of fixed point y : edgePoints = {{ 0 , y, 0 } , { 0 , y, 1 } , { 1 , y, 0 } , { 1 , y, 1 }}

Case of fixed point z : edgePoints = {{ 0 , 0 , z } , { 0 , 1 , z } , { 1 , 0 , z } , { 1 , 1 , z }}

Sides evaluation: If the target coordinate of the atom is located on lattice side (one coordinate is equal to 1 or 0 and two are not). For example, lattice side atom at point { 0 , 0 . 5 , 0 . 3 } has identical atom at point { 1 , 0 . 5 , 0 . 3 } . In this example y and z coordinates are fixed but x is exchangeable. Therefore, if the target point is represented as { x, y, z } , we can use the following formula:

where sidePoints is a set of 2 possible positions according to PBC.

Case of exchangeable point x : sidePoints = {{ 0 , y, z } , { 1 , y, z }}

Case of exchangeable point y : sidePoints = {{ x, 0 , z } , { x, 1 , z }}

Case of exchangeable point z : sidePoints = {{ x, y, 0 } , { x, y, 1 }}

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

509

(a) Target structure

(b) Prediction

Points, which don't belong to the groups above, are processed using the default loss function.

̸

Since the min( x 1 , x 2 , ..., x n ) function is undifferentiable at multiple points ( x i = x j ∀ i = j ), it makes a loss function to have a more complicated surface. Therefore, we used a norm function with order k →-∞ which is differentiable at all points as a replacement.

Therefore, overall PBC-aware loss for a structure is represented as: 503

As the count of atoms varies across different structures, the L PBC metric tends to yield 504 higher values for structures featuring a larger number of atoms. Thus, it is important to 505 normalize the loss function with the number of atoms in the structure if it would be used in 506 batches with structures with different number of atoms. Therefore, a PBC-aware loss for a 507 batch of structures is formulated as: 508

Compute resources

For our computational needs in model training and inference, we deployed a total of three 510 GPU servers with the following configurations: 511

512

513

514

515

Server 1:

GPU: NVIDIA A100/80G

CPU: 8vCPU of Intel(R) Xeon(R) Gold 6248R @ 3.00 GHz

RAM: 64Gb

Server 2: 516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

GPU: NVIDIA V100 (32GB) · CPU: 8vCPU of Intel(R) Xeon(R) Gold 6278C @ 2.60 GHz · RAM: 64Gb

Server 3:

GPU: NVIDIA V100 (32GB) · CPU: 8vCPU of Intel(R) Xeon(R) Gold 6278C @ 2.60 GHz

RAM: 64Gb

Every model training time consumed up to 2 weeks employing computing power of one GPU server.

For the ab-initio calculations implemented in VASP, we deployed a total of 5 identical CPU servers with the following configurations:

CPU: 64vCPU of Intel(R) Xeon(R) Gold 6278C CPU @ 2.60GHz

RAM: 256Gb

Structure relaxation with VASP for all six experiments mentioned in Table 3 took more that 180 thousand CPU hours. Computing power of all CPU servers was employed.

532

533

534

535

536

537

538

539

540

541

542

543

544

545

546

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

563

564

565

566

567

568

569

570

571

572

573

574

575

576

577

578

579

580

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist",

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers.

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes] ,

Justification: The main claims made in the abstract and introduction do accurately reflect the paper's contributions and scope. Every aspects mentioned in the abstract and introduction are further revealed in the main paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: All the limitations are discussed in the conclusion section. More specifically, limitations are 1) in number of atoms per unit cell of crystal structure that model can work with and 2) symmetries that our structure representation is able to encode. For example, crystals have infinite periodic structure. There is no possible way to

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA] .

Justification: Our work does not include significant theoretical results due to the fact that it is mainly focused on experiments.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

635

636

637

638

639

640

641

642

643

644

645

646

647

648

649

650

651

652

653

654

655

656

657

658

659

660

661

662

663

664

665

666

667

668

669

670

671

672

673

674

675

676

677

678

679

680

681

682

683

684

685

686

687

688

Theorems and Lemmas that the proof relies upon should be properly referenced.

Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The paper fully discloses all the information needed to reproduce the main experimental results. The information includes: 1) all hyperparameters for the models, including the random seed they were performed on 2) crystal structures used as a data, 3) VASP settings.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The raw crystal dataset can be downloaded from https://aflowlib.org. The source code for training and inferencing our models can be obtained from GitHub at https://github.com/AIRI-Institute/conditional-crystal-generation. Both data and code are also referenced in the main paper's Section 9 and Section 9 .

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips. cc/public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https://nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All the training and testing details and all the hyperparameters for the experiments are mentioned in the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Firstly, calculating statistical significance for all our experiments is computationally expensive. Secondly, experimental results(crystal structures obtained from model inference) have been approved by further ab-initio calculations implemented in VASP.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: In the paper, we state all the compute powers that we have employed for AI model training and ab-initio calculations. One can observe it in Section A.4.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research, working process, data and code correspond with NeurIPS Code of Ethics in https://neurips.cc/public/EthicsGuidelines

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The broad societal impacts are discussed int the conclusion section.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The models utilized in this research are appropriately cited within the text, including references to the AFLOW database. We have no commercial interests related to the use of these models and the crystal structure database. The code used in this study was entirely developed by our team.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The primary assets of our research are: 1) the methodology, and 2) the code for training models and performing inference. Both are thoroughly documented.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects

Guidelines: 896

897

898

899

900

901

902

903

904

905

906

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a framework for crystal structure generation, focusing on polymorphs. The framework utilizes matrix representation of crystals and various generative models used in vision tasks, with specially designed similarity metrics and loss function. It is tested on (1) modification of given structures and (2) generation from scratch within the dataset, as well as finding new structures in the Ta–W–B system.\n\nSTRENGTHS:\nThis work develops domain-specific representation, metric, and loss, so that generative models that have proven useful in image generation can apply to crystals. The topic is timely and important.\n\nWEAKNESSES:\n- In the demonstrated use cases, the generation is conditioned on elements, space group, etc., but not materials properties of interest. These show limited usefulness in materials discovery and design.\n- The matrix representation does not take physical constraints into account, e.g., space group determines symmetries in the lattice parameters. Besides, related previous works, e.g., [UniMat](https://openreview.net/forum?id=wm4WlHoXpC), should be discussed.\n- The clarity and rigorousness need to be improved (see Questions). The mathematical notations are not unified, e.g., $x$ vs $X$.\n\nQUESTIONS:\n- Line 53, “both pipelines” are not introduced. The previous paragraph is also not unclear, please check the grammar.\n- In Line 71, do “conventional limitations” mean those of conventional approaches in materials science, or previous ML/generative methods? Not just how the proposed method differs from previous ones, but also what challenge it overcomes, should be specified.\n- In the descriptions of the matrix representation, the role of elemental properties is unclear. My understanding is that they are not part of the data structure to be generated. Where and how are they used? A probably related question is, what is $el_{emb}$, and how is it obtained?\n- Line 207, what is the role of time condition t, and why does it matter for materials discovery?\n- The modification task shows significantly lower metrics than the generation task. What does this imply? Does this indicate generation is preferred over modification? If not, what are the scenarios where modification can be useful, and how to mitigate the performance?\n\nLIMITATIONS:\nDiscussed in the Conclusion.\nBesides, Sec. 9 contains a GitHub link that could break anonymity.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors studied the use of diffusion and flow matching approaches for the generation of crystalline materials. The authors trained UNet models on polymorphs in the AFLOW database (which has a series of DFT-computed properties for these materials) using either simple R3 regression, diffusion/flow matching. The authors then presented inference results on similarity to training structures (Section 3.4, which shows these methods can reproduce training structures to different extent), and showed that a subset of the modified generated crystals (with Ta, W, B) can have a small non-zero formation energy.\n\nSTRENGTHS:\n*Originality*: The authors attempted to study the problem of crystal structure generation with no invariances/equivariances other than periodic translation invariance. \n*Quality*: The authors attempted to use DFT to validate some inference results.\n*Significance*: Crystal structure generation (especially synthesizable ones) is an important problem. It seems that training from uniform noise distribution works better for CFM than training on Gaussian noise, contrary to the established results in the field.\n\nWEAKNESSES:\n*Originality*. The manuscript lacks originality. The diffusion/flow matching techniques are well-established in inorganic crystal structures (e.g. CDVAE cited here, DiffCSP/FlowMM that's not here). Sure, using a network architecture not designed for materials/crystals and using no invariances/equivalences is new, but it deviates from standard practices in the field without sufficient justification. I believe the implementations shown in the paper are a great exercise for practitioners interested in the field, but unfortunately, I do not see it as a NeurIPS paper.\n\n*Quality*. The manuscript is _very_ bare-boned, making a comprehensive technical critique challenging without appearing disproportionately critical. \n- On the ML side, there are numerous large fallacies/mistakes (e.g. no consideration of bonds between atoms at all, Sec. 3.3 there is no description of the PBC loss, the generation does not consider the unit cell, no generation with atom types, and there is no investigation of any experiments observed e.g. why is uniform noise better for CFM, the result in Table 1 appears to evaluate overfitting rather than novel generation, the list can go on). \n- On the chemistry/validation side, there are again numerous problems (why would formation energy be given during the generation process, what functional did you use in DFT, there are no comparisons against existing structures and hence cannot be claimed as novel, etc.) \n- There is no comparison against _any_ known methodologies. \n- The results overall, are very weak both in ML and in chemistry (e.g. Table 3 shows most if not all materials generated have extremely large positive formation energies despite the simple elemental composition; the remaining few negative ones are at the brink of instability, in any case they likely would not be synthesizable).  \n\n*Clarity*. The manuscript suffers from poor presentation, starting with a promotional-style title that lacks scientific descriptiveness. I unfortunately do not understand the novelty of the paper in comparison to existing methods. The paper consistently fails to provide essential explanations across both machine learning and chemical methodologies. \n- On the ML side, there are numerous things poorly presented (e.g. Figure 1 is just periodic translation invariance and in a typical manuscript would be summarized in one sentence). \n- On the chemistry side, things are greatly exaggerated (e.g. computationally making a few materials with negative formation energy can be done by undergraduate students and certainly does not warrant descriptions such as 'This significant outcome underscores the remarkable potential of our framework in uncovering thermodynamically stable materials)\n\nQUESTIONS:\nUnfortunately, without significant innovations and revisions, I do not believe I can convinced this paper would be accepted in NeurIPS.\n\nLIMITATIONS:\nPartially.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the inverse problem of generating crystal structures based on given properties, thereby avoiding the need for extensive computational resources typically required in traditional methods. The authors utilized the AFLOW materials database, selecting unstable and stable series of structures for two specific tasks: modifying structures to achieve stability and conditional structure generation.\n\nSTRENGTHS:\nThe authors experimented with various generative model approaches and evaluated two tasks in crystal generation. Additionally, they integrated the VASP software for application testing and successfully identified four previously undiscovered stable structures through conditional generation.\n\nWEAKNESSES:\n1. From the perspective of model application, although the authors used the AFLOW database for their study, they did not compare the data range and coverage with other significant databases like the Materials Project. This omission leaves a gap in understanding how the generative models perform across different datasets and whether the results are consistent and generalizable. Comparing the performance of the same generative models on different databases could provide valuable insights into the robustness and applicability of their approach.\n\n2. There is a partial break of anonymity in the GitHub link on Page 9 in this paper.\n\nQUESTIONS:\n1. In Section 2.2, the authors process the dataset for the structure modification task by training unstable structures towards stable ones. However, unlike typical trajectory datasets for structure optimization, the authors select the most stable polymorph of the same chemical formula as the target and use the remaining polymorphs as initial structures to complete the structure modification task. This dataset approach leans more towards converting between different polymorphs rather than optimizing the stability of an arbitrary atomic configuration as stated by the authors in the abstract. The authors could consider explain the relationship between using this part of the dataset and the stated objective in detail.\n\n2. In terms of the generation task, the authors generate structures by specifying formation energies. However, in Section 6, the authors only list a metric related to formation energy without providing a detailed discussion on other important aspects, such as the effectiveness of other condition controls, the validity of the generated structures, and whether duplicate structures are generated. These issues seem to remain unanswered.\n\nLIMITATIONS:\nThe authors proposed two major directions: conditional generation and conditional modification. There is room for improvement in both the experimental results and the data used for conditional modification. For example, they could consider recognizing unit cells with translational and rotational transformations and introducing more ways to assess generation results.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper deals with an important application of generative models for science: generation of crystalline structures. However, I have some serious concerns. First, scope. While comparing different methods for the same objective is informative, I am not so sure what is the purpose here. Having so many different methods certainly dilutes the main message in a short paper format like neurips. Second, approach. From what I understand, there are questionable design problems with the technical approach. Third, results. The authors spend most of their space explaining various methods such that there is little room left for explaning the impact of their results, or comparing their results with existing approaches. Finally, presentation. Figure 1 is kind of trivial or at least very simple and I am not sure it is worth a separate figure. The overall typesetting looks not too professional.\n\nSTRENGTHS:\nThe methodology selection is broad and hopefully the audience can benefit from a mini-benchmark of different generative approaches. The overall model architecture is distinctive from what I have seen in the literature.\n\nWEAKNESSES:\nI have a good number of questions on the technical approaches. More specifically, the model takes a specially formulated data structure that does not seem to be obviously invariant or equivariant under permutation, translation, rotation, which is concerning. For example, change the selection or ordering of the unit cell vectors and everything will change in an uncontrolled way.\n\nThe conditioning approach seems to be to provide desired properties as inputs to the generative approaches. I am not sure this always make sense. For example, if one desires a certain space group, there is no enforcing compliance with the space group. One can always easily check it. It is perhaps more suitable to enforce space group compliance using a guidance-based conditioning approach.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nThere is insufficient discussion about the limitations given my concerns shown above.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-14"}