Title: Towards Combinatorial Generalization for Catalysts: A Kohn-Sham Charge-Density Approach
Venue: NeurIPS 2023
Accepted: yes
Keywords: graph neural networks

equivariance

materials science

chemistry

density functional theory

combinatorial generalization

catalysts
Abstract: The Kohn-Sham equations underlie many important applications such as the discovery of new catalysts. Recent machine learning work on catalyst modeling has focused on prediction of the energy, but has so far not yet demonstrated significant out-of-distribution generalization. Here we investigate another approach based on the pointwise learning of the Kohn-Sham charge-density. On a new dataset of bulk catalysts with charge densities, we show density models can generalize to new structures with combinations of elements not seen at train time, a form of combinatorial generalization. We show that over 80% of binary and ternary test cases achieve faster convergence than standard baselines in Density Functional Theory, amounting to an average reduction of 13% in the number of iterations required to reach convergence, which may be of independent interest. Our results suggest that density learning is a viable alternative, trading greater inference costs for a step towards combinatorial generalization, a key property for applications.

Towards Combinatorial Generalization for Catalysts: A Kohn-Sham Charge-Density Approach

Abstract

The Kohn-Sham equations underlie many important applications such as the discovery of new catalysts. Recent machine learning work on catalyst modeling has focused on prediction of the energy, but has so far not yet demonstrated significant out-of-distribution generalization. Here we investigate another approach based on the pointwise learning of the Kohn-Sham charge-density. On a new dataset of bulk catalysts with charge densities, we show density models can generalize to new structures with combinations of elements not seen at train time, a form of combinatorial generalization. We show that over 80% of binary and ternary test cases achieve faster convergence than standard baselines in Density Functional Theory, amounting to an average reduction of 13% in the number of iterations required to reach convergence, which may be of independent interest. Our results suggest that density learning is a viable alternative, trading greater inference costs for a step towards combinatorial generalization, a key property for applications.

1 Introduction

The Kohn-Sham (KS) equations are a nonlinear eigenvalue problem of the form H [ ρ ]Ψ = E Ψ , where H is a symmetric diagonally dominant matrix called the Hamiltonian , Ψ is an eigenvector, E the associated eigenvalue, and ρ ( r ) = ∑ i | Ψ i ( r ) | 2 is a real-valued scalar field called the charge density , which is unknown a priori [40, 30]. The KS equations are nonlinear in the sense that the matrix H depends on the charge density ρ , which in turn depends on the eigenvectors Ψ of H . Typically it is solved by fixed-point iteration, where an initial guess for ρ is made and then a sequence of linear eigenvalue problems are solved until convergence. In the computational chemistry literature this is referred to as the self consistent field (SCF) iteration. The cost of the iteration is dominated by the eigenvalue problem [41]. Consequently methods of reducing the number of requisite iterations, e.g. with machine learning, are of great interest.

The KS equations lie at the foundation of Density Functional Theory (DFT), an approach to electronic structure theory which reformulates the N -particle quantum many-body problem in terms of an effective one-particle density [31]. DFT is widely used in a number of physico-chemical applications. One such application with important economic and environmental consequences and growing attention from the machine-learning community in recent years is the discovery of new catalysts [54].

Perhaps the greatest difficulty of catalyst discovery is the combinatorial size of the search space of structures. Even when reducing the candidate set of elements to 55 , as done in the Open Catalyst Project [8], the number of possible element combinations grows very quickly: we have ( 55 3 ) = 26 , 235 , ( 55 4 ) = 341 , 055 , and so forth. Additional factors like choice of crystal lattice, surface orientations, binding sites, and adsorbates further complicate matters, but nevertheless the number of possible element combinations is a dominating factor. Given the immensity of this search space, generalization to new combinations is a key aspect for the success of property-predicting models in applications. Put

Ca2+

Se2-

Sr2+

Te2-

Ca2+

Te2-

Sr2+

Se2-

differently, predictive models that cannot combinatorially generalize will fail to cover large parts of the search space. Moreover some authors argue that combinatorial generalization is an intrinsically important property for machine learning models in general [4].

Recent research in machine-learning for catalyst modeling aims at directly predicting the energy and/or per-atom forces on large-scale benchmarks [8, 48]. The energy is a global property of a physical system which, among other reasons of theoretical importance, helps to assess the efficiency of catalytic reactions [54]. This line of work has spawned a number of innovations in equivariant and geometric graph learning [6, 55, 12, 11]. Despite these advances, generalization performance as measured by the Energy within Threshold (EWT) metric has not achieved greater than 20% of examples in any test split at the time of this writing [1]. It remains unclear if further development or scaling will lead to practical results.

An emerging alternative to energy prediction in the context of DFT systems is the pointwise prediction of the charge density [53, 17, 5, 27, 39]. The charge density is a local rather than global property of the system: a density value is associated to each point in the computational domain. The density is fundamental in DFT in the sense that all other properties may be computed from it, e.g. the energy by way of the eigenvalue problem, potentially eliminating the need for property-specific models in favor of a single model.

In this work we carry out an empirical investigation of density based models in catalyst systems. The focus of our investigation is whether such models can generalize to new combinations of elements. Our contributions are as follows:

We compute a new large-scale DFT dataset of O (1000) relaxed bulk catalyst structures with charge densities spanning O (100 M ) unique points using the open-source DFT software Quantum Espresso and workflow software AiiDA [37, 15, 21, 22, 49, 38].

We design an evaluation methodology to measure when learned densities improve convergence versus standard baselines for density initialization in DFT. To do so we define a threshold-independent metric and also report the number of iterations saved at convergence.

We show that density models improve convergence on new structures not seen at train time. Specifically we find that learned densities outperform baselines in 83% and 86% of test cases for binary and ternary catalyst respectively. These savings amount to a reduction of 13% in the number of SCF iterations needed to reach convergence.

To the best of our knowledge, our results are the first density-based approach to show improvement over standard baselines for density initialization in DFT on new structures not seen at train time.

2 Related Work

2.1 Catalyst Modeling

The discovery of new catalysts has received significant attention from the machine-learning community in recent years. Recent work in this area has primarily been lead by the Open Catalyst Project (OCP) [8]. Building on a line of works in geometric graph learning for atomic systems [51, 42], OCP has contributed a number of modeling developments [55, 12, 11, 46, 13] and large-scale datasets [8, 48]. See Joshi et al. [26] for a summary of recent approaches based on the level of geometric information incorporated in the model.

2.2 Learning the Kohn-Sham Density

Anotable advantage of density-based approaches is that it naturally interfaces with the DFT algorithm, something not possible with energy models. Density predictions may be directly passed to the SCF cycle, which made be used to (1) better initialize a new SCF cycle versus standard baselines or (2) adaptively refine the predictions to increase the precision. In contrast the outputs of energy models are fixed and cannot be refined in this way.

Perhaps the most similar work to ours is that of Gong et al. [17], who use a graph-neural-network based approach for density prediction which encodes the local environment with a query node as we do. Using the CGGCNN [51] model, a predecessor to more advanced equivariant and geometric graph models that we use here, they perform a small-scale study of model transferability to new kinds of bonds not seen at train time, e.g. train on linear C-C-C and orthogonal C-O-C and test on orthogonal C-C-C (See Figure 7 of [17]).

Another similar work on density prediction is that of Zepeda-Núñez et al. [53], who represent an atomic structure as a set rather than a graph and define a novel network architecture with translation, rotation and permutation symmetries. They show up to four-digits of accuracy on non-catalyst test structures, including water, small organic molecules, and aluminum.

Other notable works include (1) Rackers et al. [39] who use Euclidean neural networks [14] to study generalization from smaller to larger clusters of water; (2) Brockherde et al. [5] who use a kernel-based approach to learn the dynamics of small organic molecules, Grisafi et al. [18] study transfer learning of density models from small molecules to larger ones such as hydrocarbons, (3) Jørgensen and Bhowmik [27], who show small-scale results with a message-passing network using "two-step approach" to encode the structure graph and local environment separately, in contrast to the query node approach that Gong et al. [17] and the present work uses, and (4) Schütt et al. [43] who learn elements of the Hamiltonian matrix directly for single molecule cases using a modified Schnet model [42], and show savings in SCF iterations in select cases.

2.3 Combinatorial Generalization

Combinatorial generalization (CG) is generalization from simpler data to more complex data [4]. CG and related ideas have been the subject of many machine learning works across several domains including generative models [34, 23, 33], vision and language [52, 47], reinforcement-learning [25, 7, 50], visual and analogical reasoning [20], and puzzle-solving [3].

To the best of our knowledge there are few priors works that explicitly address CG in chemistryrelated machine learning. Fernando [10] explores related ideas in the context of chemical systems. Gui et al. [19] propose an out-of-distribution graph learning benchmark with molecular tasks but do not explicitly focus on any combinatorial structure as we do here.

2.4 High-precision learning

Most modern machine learning models trained through stochastic optimization do not typically achieve training loss values around machine-precision, 10 -16 for double-precision float64 . One

notable exception is Michaud et al. [32], wherein the authors demonstrate a training method enriched with the spectrum of the Hessian for achieving near machine-precision loss performance on lowdimensional examples such as y = x 2 . It is unclear if such techniques scale to larger models . We note the distinction between high-precision at train time, a problem of optimization, and high-precision at test time, a problem of generalization . Colbrook et al. [9] argue there are fundamental limits to the precision obtainable by neural networks.

3 Methods

3.1 Modifying geometric graph learning for local property prediction

Standard geometric graph learning typically predicts a global property of a structure/molecule, e.g. the energy or band gap. Here we modify this approach for local property prediction by adding a "virtual atom" representing the query point to the graph. This approach is similar to Gong et al. [17], except our case deals with geometric graphs.

Let s = { ( R i , Z i ) } I i =1 denote a structure s with N atoms at positions R i ∈ R 3 with atomic numbers Z i . Let ρ s : R 3 → R denote the density function associated to a structure s , evaluated on a grid of query points { r j } J j =1 , with all r j ∈ R 3 .

In the usual formulation, radial graphs are typically created from each structure, i.e. draw an edge between two atoms if they lie within some radius of each other. To instead make the graph local to a point, we first adjoin the query point to the structure as a "virtual" atom ( r j , Z I +1 ) , where we interpret the atomic number Z I +1 simply as just an unused index on atom types. Note we use the same index for all query points. Letting G ( · ) denote the operation of taking the radial graph, the localized graph then takes the form G ( s ∪ ( r j , Z I +1 )) . Denote all such pairs for a structure as D s = {G ( s ∪ ( r j , Z I +1 )) , ρ s ( r j ) } . Lastly, we complete the dataset by taking a union over all structures D = ⋃ s D s .

Given this data, we proceed as usual to stochastically optimize parameters θ of a graph neural network GNN to minimize the expected loss L of samples g, ρ drawn from a (uniform) probability measure P D on the dataset D :

Since densities are continuous real-valued quantities, a regression loss is used.

3.2 Creating a dataset of catalyst charge densities with Quantum Espresso and AiiDA

Although there are emerging datasets of charge-densities [45], our evaluation requires that the settings of the data and solver be exactly the same to ensure numerical consistency. To that end, we generate our own dataset of relaxed bulk catalysts using the well-established and open-source DFT solver Quantum Espresso [16, 15] and modern computational workflow management system AiiDA [21]. In particular we utilize the relaxation workflows of Huber et al. [22], which automate the selection of computational parameters critical for achieving convergence based on expert knowledge.

For initial candidates for bulk catalysts we use the list of bulk catalysts identified by [8] from the Materials Project [24], which contains 11 , 410 structures sampled from 1 , 2 or 3 element combinations of a set of 55 elements. The total number of element combinations represented in this set is 5099 .

Grid sizes per structures can vary by up to four orders of magnitude, which can strongly skew the training data toward larger structures. To mitigate this we restrict the maximum number of atoms to be

3

12 and the maximum volume of the (conventional) cells to 200 Å . We then remove candidates with oxidation states which only occur once in the dataset, since these are not well-represented enough to evaluate, using the oxidation state guesser in pymatgen [35].

We then partition the train/val/test splits as follows. First we assign all unary catalysts remaining to the train set. Then we randomly sample binaries with oxidation states not already represented in the unaries. Next, we remove oxidation states not present in the training set from the remaining binary and ternary candidates. This preprocessing step performed to ensure all states in the val/test splits

were represented in the training set, e.g. training on H + cannot be expected to generalize to H -. Finally we assign a few binaries to the validation split and the remaining binaries and ternaries to the testing split. Finally we validate that train and test splits are disjoint by element combinations, i.e. no combination of elements in training set is repeated in the test splits.

We relax each structure in these splits using Quantum Espresso with parameters set by the AiiDA relaxation workflow [22]. We use the fast protocol with the PBE exchange-correlation functional [36] for all relaxations. Initial structure positions were first perturbed with noise. Unconverged runs were dropped from the dataset. We report the number of structures and density samples in Table 1.

3.3 Evaluating convergence of learned densities versus standard baselines in DFT

An important baseline for density based models is the "atomic charge superposition" (ACS) initialization for charge densities, commonly used in DFT solvers [2]. The ACS baseline is derived from the pre-computed collection of atomic pseudopotentials used with the DFT solver, an important technique that reduces the number of electrons in the computation. For further details see [41, 30, 31, 29].

More precisely, given a configuration of atoms { ( R i , Z i ) } I i =1 and associated atomic charge densities ρ Z i , the ACS initialization is ρ ACS 0 ( r ) = ∑ I i ρ Z i ( r -R i ) . In other words it is simply a sum of radial functions. Note this initialization corresponds to a scenario of non-interacting atoms: poor performance may be expected when the atoms interact, i.e. when there is chemistry.

In this work we consider learned densities that do not improve upon this baseline to not be of interest. From a practical point of view, clearly initializations which do not beat this baseline would not be relevant. Nevertheless, given the difficulty of obtaining high precision results with learning-based approaches, this baseline is highly non-trivial.

The degree of precision required depends on the application. One could set a threshold and check which density reached convergence in fewest iterations. However, we wish to perform the evaluations in a threshold independent way to remain agnostic to the requirements of application, similar in spirit to "area-under-the-curve" (AUC) measures popular in classification.

To that end we propose a "signed area under curves" (s-AUC) measure, essentially the area between the curves of convergence for the baseline and learned densities. See Figure 6 for examples. Note the sign is necessary because the curves may cross. More formally, if { x i } n x i =1 and { y j } n m j =1 are the SCF accuracy values for the baseline and learned curves respectively, and n = min( n x , n y ) we define

where we have used a log-scale to show rate of convergence and normalized by the pointwise max to facilitate comparison between difference curves.

We also report the percentage of iterations saved at convergence. Although this measure is threshold dependent, it has the benefit of being easily interpretable to the ML+DFT community.

3.4 Training with Spherical Channel Networks

We train a reduced sized Spherical Channel Network (SCN) for density prediction. We reduce the number of interactions to 10 and hidden channels to 256 , resulting in 35 M parameters, which is

smaller than state of the art models [55]. We use interaction cutoff to 6 . 0 Å, and a maximum number of neighbors per node of 12 . We list full hyperparameters in the Supplementary.

We train two replicates of SCNs for 420 k steps with batch size 45 across 8 GPUs. Regarding compute, for training we used approximately 2000 GPU hours for both replicates across all GPUs. We use NVIDIA A5000 cards with 24 GB of memory.

In initial experiments we also evaluated Dimenet++ [11] and GemNet [13] models but generally found performance on these models to be less than SCN models.

4 Results

4.1 Model performance

In training we obtain mean absolute errors (MAE) of O (10 -4 ) . Best validation MAE was 0 . 0011 for both replicates. One replicate was chosen to report the remainder the results. We show the distribution of MAE scores across each train/test split in Figure 2.

0

1

2

3

4

MAE

1e

3

0

25

50

Count

Test (ternary)

Test (binary)

Train (binary)

Train (unary)

4.2 Visualization of learned densities and errors

One advantage of local property prediction is the ability to directly visualize errors as a function of input points. In Figure 3 we show the true density, predicted density, and (scaled) absolute error for the top and bottom test predictions by MAE.

These visualizations allow us to inspect of the distribution of errors, which may facilitate e.g. model debugging, among other uses. Here we see a variety of patterns: ranging from nearly non-existent to quite strong errors near the core region of atoms. A closer analysis of these error patterns may inform future improvements to the dataset and model.

4.3 Evaluation of learned density convergence

We evaluate if initializing with learned densities leads to faster convergence in the SCF cycle. After predicting across the grid associated to each structure, we pass predictions to the DFT solver to initialize a new SCF cycle using exactly the same runtime parameters as the ground-truth run. Then we extract convergence information and compare them against the ground truth.

We show summary statistics of our results on SCF savings according to the s-AUC metric in Table 2. Notably we achieve positive savings in 83% of test binary cases and 86% of ternary cases, with a combined proportion of 86% . The mean and median savings in each case are positive.

In addition to the s-AUC metric, we also report savings in terms of the relative percentage of iterations saved at convergence in Table 3. The percentage of test structures with positive savings was 71% and 75% of binary and ternary test cases. The mean percentage of iterations saved was 13% for both binary and ternary splits.

We show the distribution of s-AUC scores and iteration savings across splits in Figures 4 and 5.

Wplot the convergence behavior of top test cases by s-AUC in Figure 6. Positive savings can be seen when blue-dashed curves lie beneath the red-solid lines.

MoS4W

MAE=0.00016

mp-1023954

True Density

Predicted Density

100*Error

True Density

Predicted Density

40*Error

Fe6Re2

mp-865212

MAE=0.00408

0.0

2.5

Train (unary)

0

25

Train (binary)

0

25

Test (binary)

0.5

0.0

0.5

1.0

1.5

2.0

2.5

3.0

s-AUC

0

100

Test (ternary)

Count

0

10

Train (unary)

0

20

Train (binary)

0

50

Test (binary)

20

0

20

40

60

80

Percent of Iterations Saved

0

100

Test (ternary)

Counts

0

10

20

10

7

10

1

NiRu

mp-1219922

0

10

20

Si2Ti2Y2

mp-22179

0

10

20

30

FeMnPd6

mp-1221696

0

8

16

Ni2TiZn

mp-1216672

0

10

20

MoP2Ti

mp-1216678

0

8

16

Mn2Ni2Si4

mp-1221554

0

8

16

CrCuS2

mp-5862

0

10

20

30

Co4P4Ti4

0

8

16

CaNi5

mp-1077803

0

10

20

30

Cu6Ti4

mp-622570

GT

ML

0

10

10

7

10

1

AlNi2Y2

mp-604516

0

10

20

S6TiZr

mp-1215190

0

10

20

0

8

16

As4FeNi

mp-1224951

0

10

20

Al2TiZn

mp-1217009

0

10

20

Co2Y2

mp-8053

0

15

30

CoNiV6

mp-1216439

0

8

16

CoSbV

mp-4076

0

10

20

Te4TiZr

mp-8677

0

10

20

30

Co3Si3Ti3

0

10

20

10

7

10

1

0

10

20

Cu5Y

0

10

Bi2Sr2Zn

mp-1218415

0

10

20

As4CoFe

mp-1224972

0

10

20

Se4TaTi

mp-1217829

0

10

20

30

As3Fe3Ti3

mp-1095148

0

10

20

CoNiPt2

mp-1226004

0

10

20

Ni3Sc3Sn3

mp-1078355

0

10

20

Mn2Ni2Zr2

mp-1215234

0

10

20

Ni4P4S4

mp-505820

0

10

20

10

7

10

1

Co2Mn2Si4

mp-1221669

0

10

20

S4Ti3V

mp-1217134

0

10

20

30

Cr2Fe2P4

mp-1226245

0

8

16

As2TiV

mp-1216670

0

10

20

As2Ni2Sr

0

10

20

As4Cr2Mo2

0

8

16

Al2Ni4Si2

mp-1228043

0

10

20

Cr2Fe2Ge4

mp-1226385

0

8

16

MnNiSb2

mp-1221333

0

10

20

Co2Cr2Si4

mp-1226249

4.4 Validating learned densities

We perform two validations to check if learned densities converge to the same solution as the ground truth.

First we checked for any bias in converged energy values between ground-truth and learned densities. We find almost zero difference these energy values. On the test set we found 88% of structures had exactly zero difference in energy values. Of the cases with non-zero error, the relative error on average was 1 . 2 × 10 -11 and at maximum was 1 . 5 × 10 -9 .

Second, we check if a downstream property computed from the learned densities matches their ground-truth counterparts. We compute and compare the band gaps computed from the ground-truth and ML densities using the built-in tool in ASE [28]. We find 95% of values to be exactly the same, with maximum relative error at 7 . 8 × 10 -5 .

5 Discussion

Overall we find in over 80% of test cases learned densities achieve faster convergence versus the baseline. Importantly these test cases are out-of-distribution, involving structures with new combinations of elements not seen at train time. Our results show that training on simpler structures, i.e. unary and binary, can lead to generalization on more complex structures, i.e. ternary.

6 Limitations

In this work we only test generalization for combinations of elements, and ignore stoichiometric ratios. In addition, we only investigate bulk catalysts, not the full complexity of catalyst systems as modeled in Chanussot et al. [8] which includes adsorbates, surfaces, and non-equilibrium points. Extending our analysis to these more complex systems is left to future work.

Our energy values are not directly comparable with those of Chanussot et al. [8], because we generated our dataset with a different DFT solver and pseudo-potentials. Systematic numerical differences may exist between different solvers, which makes comparison of energy values difficult.

Another limitation is that we do not report timings of our method versus traditional CPU-bound methods. Because of the high GPU-bound inference costs of our method, we do not expect timing improvement versus traditional methods. Sufficiently many GPUs will help mitigate this, as well as recent hardware development trends towards larger GPU memory.

7 Conclusion

We generated a new dataset of bulk catalyst relaxations with charge densities using open-source software. We trained a density-model pointwise on unary and binary catalysts. We showed this model can generalize to new binary and ternary catalysts, a form of combinatorial generalization. We found learned densities lead to faster convergence than standard baselines used to initialize the SCF cycle of DFT in over 80% of cases, amounting to an average of 13% saving in iterations.

7.1 Future Work

There are several directions for improvement to this work. The first is improving the proportion of test samples with positive savings, where clearly higher is better. In addition, expanding the class of structures tested, e.g. to quaternary structures and beyond, is of keen interest in applications.

A second direction is improving the rate of convergence of the learned densities. At present a number of SCF iterations are still necessary to reach high levels of convergence e.g. 10 -9 . The ideal case would be to converge rapidly to a single or few SCF iterations.

Rapid convergence across many kinds of structures opens up the interesting possibility of an end-toend hybrid learning/numerical model which first precisely predicts the density then computes the energy in one or a few SCF iterations. Such models may have different Pareto curve properties than current energy-based approaches, and may be useful in applications with high precision requirements.

8 Broader Impact

One way this work may have positive broader impact is through reducing costs of DFT computations. For example, supposing that the USA National Labs spend $100M on DFT per year, and a negligible cost of inference, then a 13% reduction in SCF iterations would yield a savings of $13M 1 . Although there are a number of practical details to resolve, we believe this work is an important first step.

The potential impacts of new catalysts discoveries may have significant economic and environmental consequences. Positive impacts may include accelerating the transition towards renewable energy, an important goal for future societies. Negative impacts may include unintended consequences such as contributing to unsustainable population growth. In addition, as with any new technology with potentially large impacts in the economy, there is risk that control of the technology may concentrate in the hands of the few and limit the distribution of benefits.

In early stages of research is difficult to discern if the aims will come to fruition. The computational discovery of new catalysts does not necessarily imply that the results are translatable into real-world materials, e.g. not all materials are known how to be made easily or economically. If such technology can be discovered, we believe that the benefits can be more equitably distributed by making this research completely open-source and accessible to global researchers. Our use of open-source DFT solvers is one notable step towards this goal over previous approaches [8].

References

Open Catalyst Project - Public Leaderboard OC20. URL https://opencatalystproject.org/ leaderboard.html .

Quantum Espresso - pw.x input description: startingpot. URL https://www.quantum-espresso.org/ Doc/INPUT_PW.html#idm899 .

David G. T. Barrett, Felix Hill, Adam Santoro, Ari S. Morcos, and Timothy Lillicrap. Measuring abstract reasoning in neural networks, 2018.

Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01261 , 2018.

Felix Brockherde, Leslie Vogt, Li Li, Mark E. Tuckerman, Kieron Burke, and Klaus-Robert Müller. Bypassing the Kohn-Sham equations with machine learning. Nature Communications , 8(1):872, October 2017. ISSN 2041-1723. doi: 10.1038/s41467-017-00839-3. URL https://www.nature.com/articles/ s41467-017-00839-3 . Number: 1 Publisher: Nature Publishing Group.

Michael M. Bronstein, Joan Bruna, Taco Cohen, and Petar Veliˇ ckovi´ c. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges, 2021.

Michael Chang, Alyssa L. Dayan, Franziska Meier, Thomas L. Griffiths, Sergey Levine, and Amy Zhang. Neural constraint satisfaction: Hierarchical abstraction for combinatorial generalization in object rearrangement, 2023.

Lowik Chanussot, Abhishek Das, Siddharth Goyal, Thibaut Lavril, Muhammed Shuaibi, Morgane Riviere, Kevin Tran, Javier Heras-Domingo, Caleb Ho, Weihua Hu, Aini Palizhati, Anuroop Sriram, Brandon Wood, Junwoong Yoon, Devi Parikh, C. Lawrence Zitnick, and Zachary Ulissi. Open Catalyst 2020 (OC20) Dataset and Community Challenges. ACS Catalysis , 11(10):6059-6072, May 2021. doi: 10.1021/acscatal.0c04525. URL https://doi.org/10.1021/acscatal.0c04525 . Publisher: American Chemical Society.

Matthew J. Colbrook, Vegard Antun, and Anders C. Hansen. The difficulty of computing stable and accurate neural networks: On the barriers of deep learning and smalersquo;s 18th problem. Proceedings of the National Academy of Sciences , 119(12), Mar 2022. doi: 10.1073/pnas.2107151119. URL https: //doi.org/10.1073%2Fpnas.2107151119 .

Chrisantha Fernando. Design for a darwinian brain: Part 1. philosophy and neuroscience, 2013.

Johannes Gasteiger, Janek Groß, and Stephan Günnemann. Directional message passing for molecular graphs. arXiv preprint arXiv:2003.03123 , 2020.

Johannes Gasteiger, Florian Becker, and Stephan Günnemann. Gemnet: Universal directional graph neural networks for molecules. Advances in Neural Information Processing Systems , 34:6790-6802, 2021.

Johannes Gasteiger, Muhammed Shuaibi, Anuroop Sriram, Stephan Günnemann, Zachary Ulissi, C. Lawrence Zitnick, and Abhishek Das. Gemnet-oc: Developing graph neural networks for large and diverse molecular simulation datasets, 2022.

Mario Geiger and Tess Smidt. e3nn: Euclidean neural networks, 2022. URL https://arxiv.org/abs/ 2207.09453 .

P. Giannozzi, O. Andreussi, T. Brumme, O. Bunau, M. Buongiorno Nardelli, M. Calandra, R. Car, C. Cavazzoni, D. Ceresoli, M. Cococcioni, N. Colonna, I. Carnimeo, A. Dal Corso, S. de Gironcoli, P. Delugas, R. A. DiStasio, A. Ferretti, A. Floris, G. Fratesi, G. Fugallo, R. Gebauer, U. Gerstmann, F. Giustino, T. Gorni, J. Jia, M. Kawamura, H.-Y. Ko, A. Kokalj, E. Küçükbenli, M. Lazzeri, M. Marsili, N. Marzari, F. Mauri, N. L. Nguyen, H.-V. Nguyen, A. Otero-de-la Roza, L. Paulatto, S. Poncé, D. Rocca, R. Sabatini, B. Santra, M. Schlipf, A. P. Seitsonen, A. Smogunov, I. Timrov, T. Thonhauser, P. Umari, N. Vast, X. Wu, and S. Baroni. Advanced capabilities for materials modelling with Quantum ESPRESSO. Journal of Physics: Condensed Matter , 29(46):465901, October 2017. ISSN 0953-8984. doi: 10. 1088/1361-648X/aa8f79. URL https://doi.org/10.1088/1361-648x/aa8f79 . Publisher: IOP Publishing.

Paolo Giannozzi, Stefano Baroni, Nicola Bonini, Matteo Calandra, Roberto Car, Carlo Cavazzoni, Davide Ceresoli, Guido L Chiarotti, Matteo Cococcioni, Ismaila Dabo, Andrea Dal Corso, Stefano de Gironcoli, Stefano Fabris, Guido Fratesi, Ralph Gebauer, Uwe Gerstmann, Christos Gougoussis, Anton Kokalj, Michele Lazzeri, Layla Martin-Samos, Nicola Marzari, Francesco Mauri, Riccardo Mazzarello, Stefano Paolini, Alfredo Pasquarello, Lorenzo Paulatto, Carlo Sbraccia, Sandro Scandolo, Gabriele

Sclauzero, Ari P Seitsonen, Alexander Smogunov, Paolo Umari, and Renata M Wentzcovitch. QUANTUM ESPRESSO: a modular and open-source software project for quantum simulations of materials. Journal of Physics: Condensed Matter , 21(39):395502, September 2009. ISSN 0953-8984, 1361-648X. doi: 10.1088/0953-8984/21/39/395502. URL https://iopscience.iop.org/article/10.1088/ 0953-8984/21/39/395502 .

Sheng Gong, Tian Xie, Taishan Zhu, Shuo Wang, Eric R. Fadel, Yawei Li, and Jeffrey C. Grossman. Predicting charge density distribution of materials using a local-environment-based graph convolutional network. Physical Review B , 100(18):184103, November 2019. doi: 10.1103/PhysRevB.100.184103. URL https://link.aps.org/doi/10.1103/PhysRevB.100.184103 . Publisher: American Physical Society.

Andrea Grisafi, Alberto Fabrizio, Benjamin Meyer, David M. Wilkins, Clemence Corminboeuf, and Michele Ceriotti. Transferable Machine-Learning Model of the Electron Density. ACS Central Science , 5 (1):57-64, January 2019. ISSN 2374-7943. doi: 10.1021/acscentsci.8b00551. URL https://doi.org/ 10.1021/acscentsci.8b00551 . Publisher: American Chemical Society.

Shurui Gui, Xiner Li, Limei Wang, and Shuiwang Ji. Good: A graph out-of-distribution benchmark. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 2059-2073. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/ 0dc91de822b71c66a7f54fa121d8cbb9-Paper-Datasets_and_Benchmarks.pdf .

Felix Hill, Adam Santoro, David Barrett, Ari Morcos, and Timothy Lillicrap. Learning to make analogies by contrasting abstract relational structure. In International Conference on Learning Representations , 2019. URL https://openreview.net/forum?id=SylLYsCcFm .

Sebastiaan P Huber, Spyros Zoupanos, Martin Uhrin, Leopold Talirz, Leonid Kahle, Rico Häuselmann, Dominik Gresch, Tiziano Müller, Aliaksandr V Yakutovich, Casper W Andersen, et al. Aiida 1.0, a scalable computational infrastructure for automated reproducible workflows and data provenance. Scientific data , 7 (1):300, 2020.

Sebastiaan P Huber, Emanuele Bosoni, Marnik Bercx, Jens Bröder, Augustin Degomme, Vladimir Dikan, Kristjan Eimre, Espen Flage-Larsen, Alberto Garcia, Luigi Genovese, et al. Common workflows for computing material properties using different quantum engines. npj Computational Materials , 7(1):136, 2021.

Geonho Hwang, Jaewoong Choi, Hyunsoo Cho, and Myungjoo Kang. MAGANet: Achieving combinatorial generalization by modeling a group action. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 14237-14248. PMLR, 23-29 Jul 2023. URL https://proceedings.mlr.press/v202/hwang23b.html .

Anubhav Jain, Shyue Ping Ong, Geoffroy Hautier, Wei Chen, William Davidson Richards, Stephen Dacek, Shreyas Cholia, Dan Gunter, David Skinner, Gerbrand Ceder, et al. Commentary: The materials project: A materials genome approach to accelerating materials innovation. APL materials , 1(1):011002, 2013.

Yiding Jiang, Shixiang Shane Gu, Kevin P Murphy, and Chelsea Finn. Language as an abstraction for hierarchical deep reinforcement learning. Advances in Neural Information Processing Systems , 32, 2019.

Chaitanya K. Joshi, Cristian Bodnar, Simon V. Mathis, Taco Cohen, and Pietro Liò. On the expressive power of geometric graph neural networks, 2023.

Peter Bjørn Jørgensen and Arghya Bhowmik. Deepdft: Neural message passing network for accurate charge density prediction, 2020.

Ask Hjorth Larsen, Jens Jørgen Mortensen, Jakob Blomqvist, Ivano E Castelli, Rune Christensen, Marcin Dułak, Jesper Friis, Michael N Groves, Bjørk Hammer, Cory Hargus, Eric D Hermes, Paul C Jennings, Peter Bjerre Jensen, James Kermode, John R Kitchin, Esben Leonhard Kolsbjerg, Joseph Kubal, Kristen Kaasbjerg, Steen Lysgaard, Jón Bergmann Maronsson, Tristan Maxson, Thomas Olsen, Lars Pastewka, Andrew Peterson, Carsten Rostgaard, Jakob Schiøtz, Ole Schütt, Mikkel Strange, Kristian S Thygesen, Tejs Vegge, Lasse Vilhelmsen, Michael Walter, Zhenhua Zeng, and Karsten W Jacobsen. The atomic simulation environment-a python library for working with atoms. Journal of Physics: Condensed Matter , 29(27):273002, 2017. URL http://stacks.iop.org/0953-8984/29/i=27/a=273002 .

Lin Lin and Jianfeng Lu. A mathematical introduction to electronic structure theory . SIAM, 2019.

Lin Lin, Jianfeng Lu, and Lexing Ying. Numerical methods for Kohn-Sham density functional theory. Acta Numerica , 28:405-539, May 2019. ISSN 0962-4929, 1474-0508. doi: 10.1017/S0962492919000047. URL https://www.cambridge.org/core/journals/ acta-numerica/article/numerical-methods-for-kohnsham-density-functional-theory/ 755DFB88349DD5F1EE1E360AD61661BF . Publisher: Cambridge University Press.

Richard M Martin. Electronic structure: basic theory and practical methods . Cambridge university press, 2020.

Eric J. Michaud, Ziming Liu, and Max Tegmark. Precision machine learning. Entropy , 25(1), 2023. ISSN 1099-4300. doi: 10.3390/e25010175. URL https://www.mdpi.com/1099-4300/25/1/175 .

Milton L Montero, Jeffrey S Bowers, Rui Ponte Costa, Casimir JH Ludwig, and Gaurav Malhotra. Lost in latent space: Disentangled models and the challenge of combinatorial generalisation. arXiv preprint arXiv:2204.02283 , 2022.

Milton Llera Montero, Casimir JH Ludwig, Rui Ponte Costa, Gaurav Malhotra, and Jeffrey Bowers. The role of disentanglement in generalisation. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id=qbH974jKUVy .

Shyue Ping Ong, William Davidson Richards, Anubhav Jain, Geoffroy Hautier, Michael Kocher, Shreyas Cholia, Dan Gunter, Vincent L Chevrier, Kristin A Persson, and Gerbrand Ceder. Python materials genomics (pymatgen): A robust, open-source python library for materials analysis. Computational Materials Science , 68:314-319, 2013.

John P. Perdew, Kieron Burke, and Matthias Ernzerhof. Generalized gradient approximation made simple. Phys. Rev. Lett. , 77:3865-3868, Oct 1996. doi: 10.1103/PhysRevLett.77.3865. URL https: //link.aps.org/doi/10.1103/PhysRevLett.77.3865 .

Giovanni Pizzi, Andrea Cepellotti, Riccardo Sabatini, Nicola Marzari, and Boris Kozinsky. AiiDA: automated interactive infrastructure and database for computational science. Computational Materials Science , 111:218-230, January 2016. ISSN 0927-0256. doi: 10.1016/j.commatsci.2015.09.013. URL https://www.sciencedirect.com/science/article/pii/S0927025615005820 .

Gianluca Prandini, Antimo Marrazzo, Ivano E Castelli, Nicolas Mounet, and Nicola Marzari. Precision and efficiency in solid-state pseudopotential calculations. npj Computational Materials , 4(1):72, 2018.

Joshua A. Rackers, Lucas Tecot, Mario Geiger, and Tess E. Smidt. Cracking the quantum scaling limit with machine learned electron densities, 2022.

Yousef Saad. Numerical methods for large eigenvalue problems: revised edition . SIAM, 2011.

Yousef Saad, James R. Chelikowsky, and Suzanne M. Shontz. Numerical Methods for Electronic Structure Calculations of Materials. SIAM Review , 52(1):3-54, January 2010. ISSN 0036-1445, 1095-7200. doi: 10.1137/060651653. URL http://epubs.siam.org/doi/10.1137/060651653 .

K. T. Schütt, H. E. Sauceda, P.-J. Kindermans, A. Tkatchenko, and K.-R. Müller. SchNet - A deep learning architecture for molecules and materials. The Journal of Chemical Physics , 148(24):241722, June 2018. ISSN 0021-9606. doi: 10.1063/1.5019779. URL https://aip.scitation.org/doi/10.1063/ 1.5019779 . Publisher: American Institute of Physics.

K. T. Schütt, M. Gastegger, A. Tkatchenko, K. R. Müller, and R. J. Maurer. Unifying machine learning and quantum chemistry - a deep neural network for molecular wavefunctions, 2019.

Jimmy-Xuan Shen. mp-pyrho, October 2022. URL https://doi.org/10.5281/zenodo.7227448 . If you use this software, please cite it using the metadata from this file.

Jimmy-Xuan Shen, Jason M. Munro, Matthew K. Horton, Patrick Huck, Shyam Dwaraknath, and Kristin A. Persson. A representation-independent electronic charge density database for crystalline materials, 2021.

Muhammed Shuaibi, Adeesh Kolluru, Abhishek Das, Aditya Grover, Anuroop Sriram, Zachary Ulissi, and C. Lawrence Zitnick. Rotation Invariant Graph Neural Networks using Spin Convolutions. arXiv:2106.09575 [cs] , June 2021. URL http://arxiv.org/abs/2106.09575 . arXiv: 2106.09575.

Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic compositionality, 2022.

Richard Tran, Janice Lan, Muhammed Shuaibi, Brandon M. Wood, Siddharth Goyal, Abhishek Das, Javier Heras-Domingo, Adeesh Kolluru, Ammar Rizvi, Nima Shoghi, Anuroop Sriram, Fé lix Therrien, Jehad Abed, Oleksandr Voznyy, Edward H. Sargent, Zachary Ulissi, and C. Lawrence Zitnick. The open catalyst 2022 (OC22) dataset and challenges for oxide electrocatalysts. ACS Catalysis , 13(5):3066-3084, feb 2023. doi: 10.1021/acscatal.2c05426. URL https://doi.org/10.1021%2Facscatal.2c05426 .

Martin Uhrin, Sebastiaan P Huber, Jusong Yu, Nicola Marzari, and Giovanni Pizzi. Workflows in aiida: Engineering a high-throughput, event-based engine for robust and modular computational workflows. Computational Materials Science , 187:110086, 2021.

Marin Vlastelica, Michal Rolinek, and Georg Martius. Neuro-algorithmic policies enable fast combinatorial generalization. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 10575-10585. PMLR, 18-24 Jul 2021. URL https://proceedings.mlr.press/v139/vlastelica21a.html .

Tian Xie and Jeffrey C. Grossman. Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties. Phys. Rev. Lett. , 120:145301, Apr 2018. doi: 10.1103/ PhysRevLett.120.145301. URL https://link.aps.org/doi/10.1103/PhysRevLett.120.145301 .

Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it?, 2023.

Leonardo Zepeda-Núñez, Yixiao Chen, Jiefu Zhang, Weile Jia, Linfeng Zhang, and Lin Lin. Deep Density: Circumventing the Kohn-Sham equations via symmetry preserving neural networks. Journal of Computational Physics , 443:110523, October 2021. ISSN 0021-9991. doi: 10.1016/j.jcp.2021.110523. URL https://www.sciencedirect.com/science/article/pii/S0021999121004186 .

C. Lawrence Zitnick, Lowik Chanussot, Abhishek Das, Siddharth Goyal, Javier Heras-Domingo, Caleb Ho, Weihua Hu, Thibaut Lavril, Aini Palizhati, Morgane Riviere, Muhammed Shuaibi, Anuroop Sriram, Kevin Tran, Brandon Wood, Junwoong Yoon, Devi Parikh, and Zachary Ulissi. An Introduction to Electrocatalyst Design using Machine Learning for Renewable Energy Storage. arXiv:2010.09435 [cond-mat] , October 2020. URL http://arxiv.org/abs/2010.09435 . arXiv: 2010.09435.

C. Lawrence Zitnick, Abhishek Das, Adeesh Kolluru, Janice Lan, Muhammed Shuaibi, Anuroop Sriram, Zachary Ward Ulissi, and Brandon M Wood. Spherical channels for modeling atomic interactions. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/forum?id=5Z3GURcqwT .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new learning framework for using machine learning to approximate chemical simulations. While many papers focus on direct property prediction (e.g. energy, forces) when applying machine learning to chemical simulation, this paper motivates the prediction of charge-density which holds promise for generalizable models across different materials systems. Following the introduction and motivation of the problem, the paper then describes related work with relevant methods in equivariant graph neural networks, materials property prediction and model generalization. Next, the authors describe their method for generating a charge-density based DFT dataset using an open-source computational chemistry engine and provide relevant statistics for their dataset, as well as a new metric (s-AUC) to measure modeling performance. The authors then describe the results of training a modestly sized SCN network on their dataset and provide an analysis of the SCN model's performance as part of the charge density prediction workflow compared to the currently applied method in Quantum Espresso calculations. Overall, the results indicate that the SCN can in some cases similar precision to the original method while being significantly more compute efficient.\n\nSTRENGTHS:\nThe paper has the following strengths:\n* Originality: The paper proposes a new learning target for computational chemistry simulations, charge-density, that is motivated by a deep understanding of the physics and chemistry of the Kohn-Sham equations. Based on authors' review, there is few prior works addressing this question even though proper models could lead to greater generalizability.\n* Quality: The authors define a new performance metric (s-AUC), define a new dataset to ensure consistency and measure modeling error rates to a high precision.\n* Clarity: The performance metrics are generally well defined and the plots in Figure 4 and Figure 5 provide a good overview of the SCN model's performance.\n* Significance: Properly training charge-density models can be significant for the application of machine learning models to computational chemistry. The problem statement has good originality and significance.\n\nWEAKNESSES:\nThe paper could be improved by:\n* Providing greater summary statistics of the SCN model's performance across the entire dataset studied. While Figure 4 and Figure 5 are useful, it is currently hard to understand how well SCN performs across the entire dataset. Table 2 provides some statistics, but the Figure caption is not informative in trying to understand the content of the Table. [Clarity, Quality]\n* It would be nice to see how effective charge-density calculation can lead to the prediction of properties found in other studies. The authors mention some reference in the text, but it would be nice to see some of those demonstrated in the paper and thereby showcase how the two methods compare. [Significance]\n* The authors mention the release of related code and data in the supplementary material. Given that the dataset and workflow are the primary contributions of this work, it would be nice to see that clarified in the main paper. [Clarity]\n* It would have been nice to see the evaluation of additional models on the generated dataset or at least an explanation as to why they were not considered. [Quality]\n* Additional data related to compute efficiency increases of the proposed method would help make a stronger case for the potential benefits of the method. [Significance]\n\nQUESTIONS:\nSome questions for the authors:\n* Can you provide additional detail on the types of materials systems your SCN model performed well on in addition to the binary and ternary classification? It could be interesting to look at generalization from the materials construction perspective.\n* Can you provide additional details for the potential compute benefits of using SCN for charge density prediction?\n* The high-precision learning motivation you have in Section 2.4 is quite relevant to this paper. Could you motivate this further in terms of high precision is needed. This might tie into applying these models in the computational workflow that you briefly mention, which would also be nice to expand upon.\n\nLIMITATIONS:\nThe authors provide a discussion of limitations in their paper, which list a set of relevant points. It would be nice to include a discussion on other types of open-source solvers, i.e. not Quantum Espresso, as well.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the researchers created a novel dataset of bulk catalyst relaxations with charge densities using open-source software. They aimed to investigate the phenomenon of combinatorial generalization, which refers to the ability of a model to effectively perform on new combinations of elements that were not seen during training. To evaluate the performance of their model, the researchers proposed a metric based on the comparison of convergence rates between their approach and standard charge-density initialization baselines. This metric served as an indicator of the quality of the results obtained. The experimental results demonstrated evidence of combinatorial generalization, indicating that the model performed well on new combinations of elements that were unseen during training. The proposed metric showed that the learned charge densities produced better results than the standard baseline in over 50% of the cases.\n\nSTRENGTHS:\nThe strengths of this paper are:\n\n* Contribution done in the field of catalysis.\n* Compute a new medium-scale DFT dataset of relaxed bulk catalyst structures with charge densities.\n* Proposal of a new metric used to verify the quality of results.\n\nWEAKNESSES:\n* The reference to figures is confusing. In lines 195 and 196 you write \" ...Figures 4.2 and  4.2 respectively.\". Same in lines 210 and 211. \n* There is no discussion regarding Figures 4 and 5 and the results they represent. \n* Although the code is not provided there is no high-level explanation of the architecture of the model.\n\nQUESTIONS:\n* What is the loss used for the proposed method?\n* Are the results reproducible? \n* Can the new proposed dataset be used by different approaches or is specific to the proposed method?\n\nLIMITATIONS:\nWhile the paper presents a novel approach and introduces a new dataset, one limitation is the absence of the dataset itself and the code required to replicate the results and gain a deeper understanding of the proposed method. The availability of the dataset and code is crucial for reproducibility, as it allows us to verify the findings. Therefore, it is essential to consider releasing the dataset and code to facilitate transparency and reproducibility.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- The authors investigate charge density modeling in catalyst systems. They generated a dataset using Quantum Espresso on Materials Project, defined a metric of SCF convergence savings, and tested models, focusing on generalization and comparing it to ACP initialization. They used GNNs with virtual atoms as query points. With reduced size SCN, 45% of binary cases and 63% of ternary cases achieved positive savings.\n\nSTRENGTHS:\n- This tackles an important problem and is a good complement to all the models directly predicting properties like energies and forces\n- The new dataset will be useful in this field\n- The paper is clearly written and easy to follow\n\nWEAKNESSES:\n- Only 50% of samples achieving positive savings, and binaries doing worse, do not seem like great results or something usable in practice.\n- More comprehensive analysis would be nice, e.g. why do errors occur near core regions of atoms (and is that good or bad?)\n- This paper only tests one model, which is based off an existing one, limiting the scope of contribution. Could there be different architectures better suited for charge density prediction?\n\nQUESTIONS:\n- How does performance compare between 1. exact materials (stoichiometry + crystal structure) in train, 2. elemental combinations that were seen in train (but not the same material), and 3. elements that were seen in train but combinations that were not? (Assuming you can't test on individual elements that were not seen in train). I am curious how high performance you can even get on in-distribution materials, and whether the low generalization performance is due to lack of generalization of just underfitting.\n\n- How granular is your grid? How granular does it need to be to be useful?\n- Is it possible to make it faster but wrong? i.e. how does one make sure that the converged result is still correct?\n\nLIMITATIONS:\n- The authors acknowledge several limitations as potential future work: larger of systems in dataset, larger models, catalyst systems with adsorbates in addition to bulks alone, understanding why binaries are worse than ternaries.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors build and train approximately E(3) equivariant graph neural networks based on Spherical Channel Network to learn the Kohn-Sham Charge-Density scalar field as a means of accelerating density functional theory (DFT) calculations by shortening the number of self-consistent field calculations needed to achieve a converged density. The authors are motivated by finding a generalizable approach to evaluate the energetics of the combinatorial space of binary, ternary, and beyond heterogeneous catalytic surfaces, which the authors point out is poorly predicted by current pure ML approaches. The model uses a virtual node to sample the value of the charge density at specific locations that are compared to ground truth DFT calculations computed at the level of PBE using the open-source package Quantum Espresso.\n\nSTRENGTHS:\nThe paper is extremely clear and well written, especially considering the need to present internal details of DFT to a general ML audience. The authors emphasize that combinatorial generalization, the ability of a model to generalize to catalytic surfaces that have combinatorial many possible substitution of constituent atoms, is a necessary metric to satisfy if ML will have impact in the heterogeneous catalysis space.\n\nWEAKNESSES:\nThe largest weaknesses of the paper are that the charge density predictions are made on a relatively few and specific set of structures (it is unclear how well this approach will generalize to more diverse structures and atom type distributions) and that results are not presented in a way makes it easy to assess the impact of the results on the broader ML+DFT community.\n\nFor this reason, I give a “weak reject” but very much hope the authors will continue to pursue this line of work as I believe it to be a powerful approach to aiding materials design. I would also consider changing my rating if the authors can quantify the benefits of their approach more strongly. I give more details below in the Questions section.\n\nQUESTIONS:\nBelow I give concrete feedback on specific parts of the paper section by section.\n\n> Introduction\nVery clear and well written. A concise and thorough description of the problem statement and goals of the paper.\n\n> Related Work\nAnother reference that could be included in related work is the following, which explicitly discusses using ML predicted charge densities of molecules as starting points for calculations:\nhttps://www.nature.com/articles/s41467-019-12875-2\nIn that paper they found minimal advantage to using ML charge densities but they only explored this for a small number of molecules (possibly only 1). Do your results align or disagree with their findings?\n\n> Methods\n>> 3.1\nI would be helpful to discuss further how the query points are selected. Additionally, it would be helpful to discuss the pros and cons of using a query point vs predicting basis functions (either plane waves or linear combination of atomic orbitals). I can appreciate that the query point is most flexible but you do seem to need a very large number of samples per structure. Quantitative comparison of the different charge density representations would be helpful.\n\n>> 3.2\nOne concern I have about using small unit cells is that this significantly limits what “patterns'' of different atomic neighborhoods the model sees and limits the significance of your results – since we don’t get a sense for how much data and samples would be needed to get comparable performance across more diverse environments (even if restricted to the same lattice). There are many ways to choose sites of a lattice but if the unit cells have lattices vectors only ~4 Å long, you’re only going to be able to explore fairly high symmetry configurations.\n\nRegarding the data selection based on oxidation state, one could consider encoding the oxidation state as an input feature in such a way that the model may be able to learn to generalize. For example, one could augment each atom type with an oxidation state feature.\n\n>> 3.3\nTypo: “this baseline is high non-trivial”\n\n> Results\nThe labels of Figures 2 and 3 are very small. Please improve the label sizes, choose an appropriate number of significant figures reported for MAE and Max Error, and specify which materials (mp-id(?) Or chemical formula) are being shown.\n\n>> 4.2\nType: “of of”\n\n>> 4.3\nHow are these predictions converted to your input charge density to Quantum Espresso? My understanding is that you need to convert it into plane waves – are there any subtleties in doing this?\n\nThe paper would be improved if the authors gave more specific metrics on the impact their results imply for performing general DFT calculations. The authors could better quantize how much compute is used for DFT calculations and how much better initial charge density predictions could save in compute. This would be a more powerful metric than just the number of examples that see any improvement in SCF steps. For example, a large fraction of the USA’s National Laboratory supercomputing resources are spent computing DFT – how much would better starting points save for a given level of accuracy? How much accuracy is needed for ML charge density prediction to be practical for these facilities to adopt into workflows?\n\nSomething that might seem like a silly benchmark but may be informative is what is the likelihood of reducing the number of SCF steps with random noise on the charge density (physically motivated noise but still noise). This would give more information as to whether what you are predicting requires learning or whether it’s due to the finicky nature of SCF calculations.\n\nAn additional sanity check that I would like to see is what is the difference between the converged charge density and the energy for the standard vs. ML accelerated calculations. It’s important to check whether the introduction of an ML derived initial charge density introduces any bias in the calculation results. It may very well be that there is no difference – but it’s important to check!\n\n> Broader Impact\nHere’s another great place to discuss how many hundreds of millions of hours of compute are spent on DFT by researchers and what potential savings good charge density starting points might enable.\n\nLIMITATIONS:\nYes, the authors are very clear about the limitations of their methods and what can be improved on. They are also clear about what are open questions in the field (section 2.1) and what is and is not known. The authors come off as clear, knowledgeable, and thoughtful.\n\nThe authors are aiming to accelerate DFT calculations which are already used used broadly. They specifically address the potential concerns of making these insights more readily available.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-19"}