Towards Combinatorial Generalization for Catalysts: A Kohn-Sham Charge-Density Approach
Authors: Phil Pope
David Jacobs
Venue: NeurIPS 2023 poster (2023)
Keywords: graph neural networks
equivariance
materials science
chemistry
density functional theory
combinatorial generalization
catalysts
TL;DR: We learn the Kohn-Sham density pointwise in bulk catalyst systems and show generalization to new combinations of elements.

Abstract: The Kohn-Sham equations underlie many important applications such as the discovery of new catalysts. Recent machine learning work on catalyst modeling has focused on prediction of the energy, but has so far not yet demonstrated significant out-of-distribution generalization. Here we investigate another approach based on the pointwise learning of the Kohn-Sham charge-density. On a new dataset of bulk catalysts with charge densities, we show density models can generalize to new structures with combinations of elements not seen at train time, a form of combinatorial generalization. We show that over 80% of binary and ternary test cases achieve faster convergence than standard baselines in Density Functional Theory, amounting to an average reduction of 13% in the number of iterations required to reach convergence, which may be of independent interest. Our results suggest that density learning is a viable alternative, trading greater inference costs for a step towards combinatorial generalization, a key property for applications.

Towards Combinatorial Generalization for Catalysts: A Kohn-Sham Charge-Density Approach
Phillip Pope
David Jacobs
University of Maryland, College Park pepopeecs. umd. edu
University of Maryland, College Park dw @umd. edu
Abstract
The Kohn-Sham equations underlie many important applications such as the dis- covery of new catalysts. Recent machine learning work catalyst modeling has focused on prediction of the energy, but has so far not yet demonstrated significant out-of-distribution generalization. Here investigate another approach based on the pointwise learning of the Kohn-Sham charge-density. On a new dataset of bulk catalysts with charge densities, we show density models new structures with combinations of elements not seen at train time, a form of combinatorial generalization. We show that over 80% of binary and ternary test cases achieve faster convergence than standard baselines in Density Functional Theory, amounting to an average reduction of 13% in the number of iterations required to reach convergence, which may be of independent interest, Our results suggest that density learning a viable alternative, trading greater inference costs for step towards combinatorial generalization, a key property for applications.
generalize
Introduction
The Kohn-Sham (KS) equations nonlinear eigenvalue problem of the form Hip EN,, where H is a symmetric diagonally dominant matrix called the Hamiltonian, T is an eigenvector, E the real-valued scalar field called the charge density, which is unknown a priori [40] 30). The KS equations are nonlinear in the sense that the matrix H depends the charge density p. which in turn depends on the eigenvectors V of H. Typically is solved by fixed- fixed-point iteration, where an initial guess for p is made and then sequence of linear eigenvalue problems are solved until convergence. In the computational chemistry literature this is referred to as the self consistent field (SCF) iteration. The cost of the iteration is dominated by the eigenvalue problem [41] Consequently methods of reducing the number of requisite iterations, e.g. with machine learning, are of great interest.
associated eigenvalue, and p(r) E; V (r))²
The KS equations lie the foundation of Density Functional Theory (DFT), approach to electronic structure theory which reformulates the N-particle quantum many-body problem in terms of an effec- tive one-particle density (3I) DFT is widely used in a number of physico-chemical applications. One such application with important economic and environmental consequences and growing attention from the machine-learning community in recent years the discovery of new catalysts
[54
Perhaps the greatest difficulty of catalyst discovery is the combinatorial size the search space of structures. Even when reducing the candidate set of elements to 55, as done in the Open Catalyst
Project [8), the number of possible element combinations grows very quickly: we have (55) 26, 235, 55 341, 055, and so forth. Additional factors like choice of crystal lattice, surface orientations, binding sites, and adsorbates further complicate matters, but nevertheless the number of possible element combinations is dominating factor. Given the immensity of this search space, generalization to new combinations is key aspect for the success of property- property-prediicing models in applications. Put
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Spr
Te
Figure 1:
Simple illustration of a combinatorial generalization task with binary catalysts (CaSe, SeTe, CaTe] for training and {SrSe} for testing. Can density values of the structures on the left be used to predict the density values on the right? Note the combination of elements occurring in the test set does not occur in the training set, but its constituent elements do. Materials project IDs from left to right are mp- 1415, mp- 1958, mp-1519, and mp-2758. Structures visualized with the Materials Project web app 24 NB: density values are not shown.
differently, predictive models that cannot combinatorially generalize will fail cover large parts of the search space. Moreover some authors argue that combinatorial generalization is an intrinsically important property for machine learning models in general 4
Recent research machine-learnnng for catalyst modeling aims at directly predicting the energy and/or per-atom forces large-scale benchmarks { 48 The energy is global property of physical system which, among other reasons theoretical importance, helps to assess the efficiency of catalytic reactions [54]. This line of work has spawned number of innovations in equivariant
and geometric graph learning 6 55 12 IT]. Despite these advances, generalization performance measured by the Energy within Threshold (EWT) metric has not achieved greater than 20% of examples in any test split at the time of this writing []. remains unclear if further development or scaling will lead practical results.
emerging alternative energy prediction in the context of DFT systems the pointwise prediction of the charge density 53 17 27 39. The charge density is a local rather than global property of computational domain. The density fundamental in DFT in the sense that all other properties may be computed from it, e.g. the energy by way of the eigenvalue problem, potentially eliminating need for property-specific models in favor of single model.
system: a density value associated to each point
In this work we carry out an empirical investigation of density based models in catalyst systems. The focus of our investigation is whether such models can generalize to new combinations of elements. Our contributions are as follows:
We compute a new large-scale DFT dataset of O(100) relaxed bulk catalyst structures with charge densities spanning O (100M) unique points using the open-source DFT software Quantum Espresso and workflow software AiiDA [3
38
We design an evaluation methodology to measure when learned densities improve con- vergence versus standard baselines for density initialization in DFT. To do define threshold-inde indeeeddenndeeedennt metric and also report the number iterations saved at convergence.
We show that density models improve convergence on new structures not seen at train time. Specifically we find that learned densities outperform baselines in 83% and 86% of test cases for binary and ternary catalyst respectively. These savings amount to a reduction of 13% in the number of SCF iterations needed to reach convergence.

---

To the best of our knowledge, our results are the first density- density-based approach to show improvement over standard baselines for density inttialization in DFT on new structures not seen at train time.
2 Related Work
2.1 Catalyst Modeling
The discovery of new catalysts has received significant attention from the machine-learning commu- nity in recent years. Recent work in this area has primarily been lead by the Open Catalyst Project (OCP) []. Building on line of works in geometric graph learning for atomic systems [51 4 OCP
has contributed number of modeling developments [55] 12 46 13] and large-scale datasets 8 48). See Joshi et [26] for a summary of recent approaches based on the level of geometric
information incorporated in the model.
2.2 Learning the Kohn-Sham Density
A notable advantage of density-basee based approaches is that it naturally interfaces with the DFT algorithm, something not possible with energy models. Density predictions may be directly passed to the SCF cycle, which made be used to (1) better initialize new SCF cycle versus standard baselines or (2) adaptively refine the predictions to increase the precision. contrast the outputs of energy models are fixed and cannot be refined in this way.
is that of Gong et al. [17]. who graph- heaph-neural-networr based approach for density prediction which encodes the local environment with query node as do. Using the CGGCNN [5I] model, a predecessor more advanced equivariant and geometric graph models that we use here, they perform a small-scale study of model transferability to new kinds of bonds not seen at train time, e.g. train on linear C-C-C and orthogonal C-O-C and test on orthogonal C-C-C (See Figure 7 of [17)).
Perhaps the most similar work
Another similar work density prediction is that of Zepeda-Núñee atomic structure as a set rather than a graph and define a novel network architecture with translation, rotation and permutation symmetries. They show up four-digits of accuracy non-catalyst test. structures, including water, small organic molecules, and aluminum.
al. [53], who represent an
Other notable works include (1) Rackers al. 39 who use Euclidean neural networks [4] to study generalization from smaller to larger clusters water; (2) Brockherde et al. (5) who use kernel-based approach to learn the dynamies of small organic molecules, Grisafi et al. [18] study transfer learning of density models from small molecules to larger ones such as hydrocarbons, (3) Jørgensen and Bhowmik (27). who show small-scale results with message- passseggpssnggggg network using "two-step approach" to encode the structure graph and local environment separately, in contrast the query node approach that Gong al. [17] and the present work uses, and (4) Schütt et al. 43 who learn elements of the Hamiltonian matrix directly for single molecule cases using a modified Schnet and show savings in SCF iterations in select cases.
model 42
2.3
Combinatorial Generalization
Combinatorial generalization (CG) is generalization from simpler data to complex data 4 CG and related ideas have been the subject many machine learning works across several domains
including generative models 34 23 331. vision and language [52 47). reinforcement-leamiee
(25 [50]. visual and analogical reasoning 20] and puzzle-solving
To the best of our knowledge there are few priors works that explicitly address CG in chemistry- related machine learning. Fernando [10) explores related ideas in the context of chemical systems. Gui et al. [19] propose out-offdiistriutioo graph learning benchmark with molecular tasks but do not explicitly focus on any combinatorial structure as we do here.
2.4 High- nigh--rreisioo learning
Most modern machine learning models trained through stochastic optimization do not typically achieve training loss values around machine machine-precision. 10-16 for double- float64. One
3

---

notable exception is Michaud et al. [2], wherein the authors demonstrate a training method enriched with the spectrum of the Hessian for achieving near muchine-precistee loss performance on low- dimensional examples such as y = x². It is unclear if such techniques scale to larger models.. We note the distinction between High-precision at train time, problem of optimization, and high-precision at test time, a problem of generalization. Colbrook et al. [9] argue there are fundamental limits to the precision obtainable by neural networks.
3 Methods
3.1 Modifying geometric graph learning for local property prediction
Standard geometric graph learning typically predicts global property of structure/molecule, e.g- the energy or band gap. Here we modify this approach for local property prediction by adding a "virtual atom" representing the query point to the graph. This approach is similar to Gong et al. (17). except our case deals with geometric graphs.
Let = (R,, Zi) denote structure with N atoms at positions R, E IR3 with atomic numbers 7, Let : R3 + R denote the density function associated to a structure 8, evaluated on a grid of query points {r, {r,) with all r E R².
In the usual formulation, radial graphs are typically created from each structure, i.e. draw an edge between two atoms if they lie within some radius of each other. To instead make the graph local to a point, we first adjoin the query point to the structure as a "virtual" atom (r,, Zı+1), where we interpret the atomic number Z1+1 simply as just an unused index on atom types. Note we use the same index for all query points. Letting G(-) denote the operation of taking the radial graph, the localized graph then takes the form G(s U (rj, ZI+1)). Denote all such pairs for a structure (g(s U (r,: 71+1)),
as D.- structures D = U.
Lastly, we complete the dataset by taking union over all
Given this data, we proceed as usual to stochastically optimize parameters 0 of graph neural network GNN to minimize the expected loss L of samples drawn from (uniform) probability measure PD on the dataset D:
((GNN)(O))
min E S.pruP
Since densities are continuous real-valued quantities, regression loss is used.
3.2 Creating dataset of catalyst charge densities with Quantum Espresso and AiiDA
Although there are emerging datasets of charge-densities [55, our evaluation requires that the settings of the data and solver be exactly the same to ensure numerical consistency. To that end, we generate our own dataset of relaxed bulk catalysts using the well-established and open-source DFT solver Quantum Espresso 16 15) and modern computational workflow management system AiiDA [2I] In particular we utilize the relaxation workflows of Huber et al. (22), which automate the selection of computational parameters critical for achieving convergence based on expert knowledge.
For initial candidates for bulk catalysts we use the list of bulk catalysts identified by g from the Materials Project (24), which contains 11, 410 structures sampled from 1, or 3 element combinations of a set of 55 elements. The total number of element combinations represented in this set is 5099.
Grid sizes per structures can vary by up to four orders of magnitude, which can strongly skew the training data toward larger structures. To mitigate this we restrict the maximum number of atoms to be
12 and the maximum volume of the (conventional) cells to 200 A We then remove candidates with oxidation states which only occur once in the dataset, since these are not well-represented enough to evaluate, using the oxidation state guesser in pymatgen 35).
We then partition the train/val/test splits as follows. First we assign all unary catalysts remaining to the train set. Then we randomly sample binaries with oxidation states not already represented in the unaries. Next, we remove oxidation states not present in the training set from the remaining binary and ternary candidates. This preprocessing step performed to ensure all states in the val/test splits

---

Table I:
Sample sizes of dataset
N structures
N Namppls
Train (unary)
47
3.8M
Train (binary)
392
69M
Val (binary)
4
300k
Test (binary)
360
72M
Test (ternary)
1116
380M
were represented in the training set, e.g. training on H+ cannot be expected to generalize to H- Finally we assign few binaries to the validation split and the remaining binaries and ternaries the testing split. Finally we validate that train and test splits are disjoint by element combinations, i.c. no combination of elements training set repeated in the test splits.
We relax each structure in these splits using Quantum Espresso with parameters set by the AiiDA relaxation workflow [2]. We use the fast protocol with the PBE exchange-correlatioe functional [6 for all relaxations. Initial structure positions were first perturbed with noise. Unconverged runs were dropped from the dataset. We report the number of structures and density samples in Table I
3.3 Evaluating convergence of learned densities versus standard baselines in DFT
An important baseline for density based models the "atomic charge superposition" (ACS) initializa- tion for charge densities, commonly used in DFT solvers (2). The ACS baseline is derived from the pre-computed collection of atomic pseudopotentials used with the DFT solver, an important technique
that reduces the number of electrons in the computation. For further details see 41 30 31 29
More precisely, given a configuration of atoms {RR, Zo) and associated atomic charge densities pCCS (r) = Σ. Pz. (r R,). In other words it is simply sum of scenario of non-interacting atoms: poor
Pz., the ACS initialization is pb
radial functions. Note this initialization corresponds performance may be expected when the atoms interact, i.e. when there is chemistry.
In this work consider learned densities that do not improve upon this baseline not be of interest. From a practical point of view, clearly initializations which do not beat this baseline would not be relevant. Nevertheless, given the difficulty of obtaining high precision results with learning-hasee approaches, this baseline is highly non-trivial.
The degree of precision required depends on the application. One could set threshold and check which density reached convergence in fewest iterations. However, we wish to perform the evaluations in threshold independent way to remain agnostic to the requirements of application, similar in spirit to larea-under-the sendder-the-uurte the--crve" (AUC) measures popular in classification.
To that end we propose "signed area under curves" (s-AUC) measure, essentially the area between curves convergence for the baseline and learned densities. See Figure for examples. Note the sign is necessary because the curves may cross. More formally, if (a; ) and {uy J== are the SCF accuracy values for the bascline and learned curves respectively, and min(n., nly) define
En Lo log(z,) log(y:)
S-AUC
(2)
(axx(llo((;)))...))))))))))))))) log.).
where we have used log-scale to show rate of convergence and normalized by the pointwise max to facilitate comparison between difference curves.
We also report the percentage iterations saved convergence. Although this measure threshold dependent, it has the benefit of being easily interpretable to the ML+DFT community.
3.4 Training with Spherical Channel Networks
We train a reduced sized Spherical Channel Network (SCN) for density prediction. We reduce the number of interactions to 10 and hidden channels to 256, resulting in 35M parameters, which is
5

---

smaller than state of the art models [55]. We use interaction cutoff to 6.0A, and maximum number of neighbors per node of 12. We list full hyperparameters in the Supplementary.
We train two replicates of SCNs for 420k steps with batch size 45 across GPUs. Regarding compute, for training used approximately 2000 GPU hours for both replicates across all GPUs. We use NVIDIA A5000 cards with 24 GB of memory.
In initial experiments also evaluated Dimenet++ and GemNet [3 models but generally found performance on these models to be less than SCN models.
4 Results
4.1 Model performance
In training obtain mean absolute errors (MAE) of ((10-t). Best validation MAE was 0.0011 for both replicates. One replicate was chosen to report the remainder the results. We show the distribution of MAE scores across each train/test split in Figure 2
Test (ternary)
50
Test (binary)
Train (binary)
25
Train (unary)
MAE
1e-3
Figure 2: Distributions of MAE scores for train/test splits
4.2 Visualization of learned densities and errors
One advantage of local property prediction is the ability to directly visualize errors as function of input points. In Figure 3 we show the true density, predicted density, and (scaled) absolute error for the top and bottom test predictions by MAE.
These visualizations allow us to inspect of the distribution of errors, which may facilitate e.g. model debugging, among other uses. Here see variety of patterns: ranging from nearly non-existent quite strong errors near the core region of atoms. A closer analysis of these error patterns may inform future improvements the dataset and model.
4.3 Evaluation of learned density convergence
We evaluate if initializing with learned densities leads to faster convergence in the SCF cycle. After predicting across the grid associated to each structure, we pass predictions to the DFT solver to initialize new SCF cycle using exactly the runtime parameters the ground-truth run. Then we extract convergence information and compare them against the ground truth.
We show summary statistics of our results on SCF savings according to the S-AUC metric in Table 2 Notably we achieve positive savings 83% of test binary cases and 86% of ternary cases, with combined proportion of 86%. The mean and median savings in each case positive.
In addition to the S-AUC metric. also report savings in terms of the relative percentage of iterations saved at convergence in Table 3. The percentage of test structures with positive savings was 71% and 75% of binary and ternary test cases. The mean percentage of iterations saved was 13% for both binary and terarry splits.
We show the distribution of S-AUC scores and iteration savings across splits in Figures and 5
W plot the convergence behavior of top test cases by S-AUC Figure Positive savings can seen when bluc-dashed curves lie beneath the red-solid lines.

---

Bendty
Peeiited Bemdty
c0rEmr
0
0
Figure 3: Test densities (2D projections): ground truth, predicted, and error. Top row is low emor test structure MoS, W (mp-1023954) and bottom row is higher error test structure Feg (mp-8862m with higher MAE enror. Errors rescaled display more clearly. To visualize 3D densities are regridded to approximately a cube 44 and summed along one axis.
Table 2: Summary statistics of SCF savings with the s-AUC metric.
N
N
N+/N (%)
Mean
Median
Max
Min
Train (unary)
47
45
96
0.6
0.3
7.3
-0.1
Train (binary)
392
379
97
4.0
0.6
2.9 x 10a
-0.5
Test (binary)
360
298
83
7.8
0.2
1.9
10y
-3.7
Test (ternary)
1116
965
86
2.7
0.4
7.2 102
-2.8 10a
Table 3: Summary statistics of relative iterations saved at convergence.
N
N4
N+/N (%)
Mean (%)
Median (%)
Max (%)
Min (%)
Train (unary)
47
32
68
15
14
78
-53
Train (binary)
392
361
92
22
21
63
-50
Test (binary)
360
256
71
13
12
72
-53
Test (ternary)
1116
841
75
13
12
65
-90

---

2.5
Train (unary)
0.0
25
Train (binary)
Count
(binary)
25
0
100
Test (ternary)
0
-0.5
0.0
0.5
1.0
1.5
2.0
2.5
3.0
S-AUC
Figure 4: Distributions of S-AUC metries. Outliers dropped for visualization.
10
Train (unary)
Train (binary)
Counts
50
(binary)
0
100
(ternary)
0
-20
20
40
60
80
Figure 5: Distributions of relative iteration savings. Outliers dropped for visualization.
NIRu mp-1219922
SIATLYS mp-22219
FoMnPd: mp-1222660
NETTZZA
MoP.TI mp-1216677
Mn.Ni,Sl, mp-1221554
ErCuds mp-5862
CosP,Tl, mp-1102138
CaNi: mp- mp-1077803
CULTI mp-522570
mp-3216672
18-1
18
10
10
26
10 20 30
16
10
20
14
10 30
16
10 20
AlNigY,
5,T12r
Al,T
Al, 1i2m
Co:Y;
Cod
Te,Titr
Co,
mp-1215119
mp-1797
mp-2797
mp-1219415
mp
1224222
mp-1217829
mp-1095148
mp
mp-1226004
mp- mp-1078355
mp-2215254
mp-305520
10
10
20
16
Co:Mn.Si,
S.Tl.V
Cr.Fe,P
AIV
AsyNi,Sr
Mrg
Al,Ni,Si
CryPe:Ges
MnNiSb;
Co,Cr,Si.
mp-3221669
mp-1217134
mp-1226245
mp-1216670
mp-4916
mp-1226254
mp-3226043
mp-1226385
mp- 322133
mp-1226269
18
18
10
28
10
29
16
20
30
10
18
20
16
18
Figure 6: Top convergence results for structures by S-AUC metric. Red solid lines are ground- truth DFT convergence, blue dashed lines are learned densities. Y-axis is SCF accuracy on log-scale and x-axis number of SCF iterations. Chemical formulas and material project ids given above each subplot. Note these structures were not seen at train time.

---

4.4 Validating learned densities
We perform two validations check if learned densities converge to the same solution the ground truth.
First we checked for any bias in converged energy values between ground-truth and learned densities. We find almost zero difference these energy values. On the test set we found 88% of structures had exactly zero difference energy values. Of the cases with non-zero error, the relative error on average was 1.2 x 10-11 and at maximum was 1.5 x 10-9.
Second, we check if downstream property computed from the learned densities matches their ground-truth counterparts. We compute and compare the band gaps computed from the ground-truth and ML densities using the built-in tool in ASE (28). We find 95% of values to be exactly the same, with maximum relative error at 7.8 x 10-5.
5 Discussion
Overall we find in over 80% of test cases learned densities achieve faster convergence versus the baseline. Importantly these test cases out-of-distribution,, involving structures with combinations of elements not seen at train time. Our results show that training on simpler structures, iec unary and binary, can lead generalization on more complex structures, i.c. ternary.
Limitations
In this work only generalization for combinations of elements, and ignore stoichiometric ratios. In addition, we only investigate bulk catalysts, not the full complexity of catalyst systems modeled in Chanussot al. g which includes adsorbates, surfaces, and non-equilibrium points. Extending our analysis to these more complex systems is left to future work.
Our energy values not directly comparable with those of Chanussot al. [8], because generated our dataset with a different DFT solver and sseudo-potential. Systematic numerical differences may exist between different solvers, which makes comparison of energy values difficult.
Another limitation is that we do not report timings of our method versus traditional CPU-bound methods. Because of the high GPU-bound inference costs of our method, we do not expect timing improvement versus traditional methods. Sufficiently many GPUs will help mitigate this, as well as recent hardware development trends towards larger GPU memory.
7 Conclusion
We generated a new dataset of bulk catalyst relaxations with charge densities using open-source software. We trained a density-model pointwise on unary and binary catalysts. We showed this model generalize to new binary and ternrry catalysts, a form of combinatorial generalization. We found learned densities lead to faster convergence than standard baselines used to initialize the SCF cycle of DFT in over 80% of cases, amounting to average of 13% saving in iterations.
7.1 Future Work
There are several directions for improvement to this work. The first is improving the proportion of test samples with positive savings, where clearly higher better. addition, expanding the class of structures tested, e.g. to quaternary structures and beyond, is of keen interest in applications.
A second direction is improving the rate of convergence of the learned densities. At present number of SCF iterations are still necessary to reach high levels of convergence e.g. 10 The ideal case would be to converge rapidly to single or few SCF iterations.
Rapid convergence across many kinds of structures opens up the interesting possibility of an end-to- end hybrid learningnnmmeriaal model which first precisely predicts the density then computes the energy in one or a few SCF iterations. Such models may have different Pareto curve properties than current energy- based approaches, and may be useful in applications with high precision requirements.

---

Broader Impact
One way this work may have positive broader impact through reducing costs of DFT computations. For example, supposing that the USA National Labs spend $100M on DFT per year, and negligible cost of inference, then a 13% reduction in SCF iterations would yield a savings of $13M | Although there are a number of practical details to resolve, we believe this work is an important first step.
The potential impacts of new catalysts discoveries may have significant economic and environmental consequences. Positive impacts may include accelerating the transition towards renewable energy, an important goal for future societies. Negative impacts may include unintended consequences such as contributing to unsustainable population growth. In addition, as with any new technology with potentially large impacts in the economy, there is risk that control of the technology may concentrate in the hands of the few and limit the distribution of benefits.
In early stages of research is difficult to discern if the aims will come to fruition. The computational discovery of new catalysts does not necessarily imply that the results are translatable into real-world materials, e.g. not all materials are known how be made easily economically. If such technology can be discovered, we believe that the benefits can be more equitably distributed by making this research completely open-source and accessible global researchers. Our use of open-source DFT solvers is one notable step towards this goal over previous approaches (8).
Acknowledgements
PP thanks Professors Maria Cameron, Howard Elman, Tom Goldstein, Ramani Duraiswami, Pratyush Tiwary, and Hong-Zhou Ye, well as Dr. Ruiyu Wang for feedback on this project. PP also thanks the University of Maryland Institute for Advanced Computer Studies (UMIACS) for providing the computing infrastructure used in this project.
We thank anonymous reviewer for suggesting this way of interpreting results
10

---

References
[1] Open Catalyst Project Public Leaderboard OC20. URL http:://ppeenaaaalyyppgroe.....p leaderboard html
[2] Quantum Espresso pw.x input description: startingpot. URL http:://www.quaattu-eeeeprrpppee DoC/INPUT JINPUU_PWW.AA ht.lmtm1 #1 #idm899
[3] David G. T. Barrett, Felix Hill, Adam Santoro, Ari S. Morcos, and Timothy Lillicrap. Mcasuring abstract reasoning in neural networks, 2018.
[4] Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti. David Raposo, Adam Santoro. Ryan Faulkner. et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint 1806vv8800.02233 2018.
[5] Felix Brockherde, Leslie Vogt, Li Li, Mark E. Tuckerman. Kieron Burke, and Klaus Robert Müller. Bypass- ing the Kohn-Sham equations with machine learning. Nature Communicattinss 8(1):872, October 2017. ISSN 2041-1723. doi: 10. (033///1467-017-000000
URL https://wwww www.w...tu.....mmmm com/articles/ 041467-077-08839 Number: Publisher: Nature Publishing Group.
Michael M. Bronstein, Joan Bruna, Taco Cohen, and Petar Veličkovié. Geometric deep learning: Grids, groups, graphs, geodesies, and gauges, 2021.
[7] Michael Chang, Alyssa L. Dayan, Franziska Meier, Thomas L. Griffiths, Sergey Levine, and Amy Zhang. Neural constraint satisfaction: Hicrarchical abstraction for combinatorial generalization object rearrangement, 2023.
[8] Lowik Chanussot, Abhishek Das. Siddharth Goyal, Thibaut Lavril, Muhammed Shuaibi, Morgane Riviere, Kevin Tran, Javier Mteras-Domingg Caleb Ho, Weihua Hu, Aini Palizhati, Anuroop Sriram, Brandon Wood, Junwoong Yoon, Devi Parikh, C. Lawrence Zitmick, and Zachary Ulissi. Open Catalyst 2020 (OC20) Dataset and Community Challenges. ACS Catalysis, 1 1010))6059-6000 6072, May 2021. doi: 10.1021 facscatallll.eeee URL /ttt//ttt:://oo..... htt:///doo..rr//// org/ 10. 1111////c1aaaaa. 0c04525 Publisher: American Chemical Society.
[9] Matthew J. Colbrook, Vegard Antun, and Anders C. Hansen. The difficulty of computing stable and accurate neural networks: On the barriers of deep learning and smalersquo:s 18th problem. Proceedings of the National Academy of Sciences, 119(1 Mar 2022. doi: 11107//pas.210775111999 URL //doi orgg 10. 1077 2107151119
[10] Chrisantha Fernando. Design for darwinian brain: Part I. philosophy and neuroscience, 2013.
[11] Johannes Gasteiger, Janek GroB, and Stephan Günnemann. Directional message passing for molecular graphs. arXiv preprint arXiv:2003.03322: 2020.
[12] Johannes Gasteiger. Florian Becker, and Stephan Günnemann. Gemnet: Universal directional graph neural networks for molecules. Advances in Nearal Information Processing Systems, 34:6790-6802. 2021.
[13] Johannes Gasteiger, Muhammed Shuaibi, Anuroop Sriram, Stephan Günnemann, Zachary Ulissi, C. Lawrence Zimick, and Abhishek Das. Gemnet-oc: Developing graph neural networks for large and diverse molecular simulation datasets, 2022.
[14] Mario Geiger and Tess Smidt, e3nn: Euclidean neural networks, 2022. 094533
[15] P. Giannozzi, O. Andreussi, T. Brumme, O. Bunau, M. Buongiorno Nardelli, M. Calandra, R. Car, C. Cavazzoni, D. Ceresoli, M. Cococcioni, N. Colonna, I. Carnimco, A. Dal Corso, S. de Gironcoli, P. Delugas, R. A. DiStasio, A. Ferretti, A. Floris, G. Fratesi, G. Fugallo, R. Gebauer, U. Gerstmann, F. Giustino, T. Gorni, J. Jia, M. Kawamura, H.-Y. Ko, A. Kokalj, E. Küçükkennli M. Lazzeri, M. Marsili, L. Nguyen, H.-V. Nguyen, A. Otero-de-la Roza, L. Paulatto, S. Poncé, D. Rocca, R. Sabatini, B. Santra, M. Schlipf, A. Seitsonen, A. Smogunov. I. Timrov, T. Thonhauser, P. Umari, N. Vast, X. Wu, and S. Baroni. Advanced capabilities for materials modelling with Quantum ESPRESSO. Journal of Physics: Condensed Matter, 46546), October 2017. ISSN 0953-8984. doi: 10. URI. https://don 10g 1ww.:108/1111-1110 648x/ aa8f79 Publisher: IOP
N. Marzari, F. Mauri,
Publishing.
[16] Paolo Giannozzi, Stefano Baroni, Nicola Bonini, Matteo Calandra, Roberto Car, Carlo Cavazzoni, Davide Ceresoli, Guido L Chiarotti, Mattco Cococcioni, Ismaila Dabo, Andrea Dal Corso, Stefano de Gironcoli, Stefano Fabris, Guido Fratesi, Ralph Gebauer, Uwe Gerstmann, Christos Gougoussis, Anton Kokalj, Michele Lazzeri, Layla zarello, Stefano Paolini, Alfredo Pasquarello. Lorenzo Paulatto, Carlo Sbraccia, Sandro Scandolo, Gabriele
Nicola Marzari, Francesco Mauri, Riccardo Maz-
11

---

Sclauzero. Ari P Seitsonen, Alexander Smogunov, Paolo Umari, and Renata M Wentzcovitch. QUAN- TUM ESPRESSO: a modular and open-source software project for quantum simulations of materials.
Journal of Physies: Condensed Matter, 21(39)395500,, September 2009, ISSN 0953-8984, 1361-648X.
doi: 10. 1088/0933-----
8984/21/39/395502.
URL
1088/
0953 /984/21/39/338332222 3955552
[17] Sheng Gong, Tian Xie, Taishan Zhu, Shuo Wang, Eric R. Fadel, Yawei Li, and Jeffrey C. Grossman. Predicting charge density distribution of materials using a local-environmeet--aeee graph convolutional network. Physical Review B, 100(18): (00(11)188413, November 2019. doi: 10. 103/PhysRevB, 100. 184103.
URL htLhhtts:://aan aps wwto/oog//oooooooooo /01/ 1103/PhysRevB. 184118 Society.
Publisher: American Physical
[18] Andrea Grisafi, Alberto Fabrizio, Benjamin Meyer, David M. Wilkins, Clemence Corninbocuf, and Michele Ceriotti. Transferable Machine- Learning Model of the Electron Density, ACS Central Science, (1):57 64, January 2019. ISSN 2374-7943, doi: 10. 10.1022//aa 10211/accenttsc.. 8000551. URL //rp:/ 10. wtl.1021aacce wttp///cccnttcc...mmm 8b00551
Publisher: American Chemical Society.
[19] Shurui Gui, Xiner Li, Limei Wang, and Shuiwang Ji. Good: A graph out-of-distribution bench- mark. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors. Ad- vances Neural Information Processing Systems, volume 35, pages 2059-2072 Curran Asso- ciates, Inc.. 2022. URL
neurips. wwwrrps.ccppapppeeeeeeee cctpsper_pppppppeeeeeeeeee http:ppprr..lleeeppprr////////// 0oc51de822277166//555faaaa222222 Papeeebbbb--apar--e-ee Datasets and Benchmarks. pdf
[20] Felix Hill, Adam Santoro, David Barrett, Ari Morcos, and Timothy Lillicrap. Learning to make analogies by contrasting abstract relational structure. In International Conference on Learning Representations, 2019. URL https::/ wtwprr/reeeeeeeeeooorortoteteete
[21] Sebastiaan P Huber, Spyros Zoupanos, Martin Uhrin, Leopold Talirz, Leonid Kahle, Rico Häuseelaann, Dominik Gresch, Tiziano Müller. Aliaksandr V Yakutovich. Casper W Andersen, et al. Aïida 1.0, a scalable computational infrastructure for automated reproducible workflows and data provenance. Scientific data, 7 (1):300, 2020.
[22] Sebastiaan P Huber, Emanuele Bosoni, Marnik Berex, Jens Bröder, Augustin Degomme, Vladimir Dikan, Kristjan Eimre, Espen Flage-Larsen, Alberto Garcia, Luigi Genovese, et al. Common workflows for computing material properties using different quantum engines. npj Computational Materials, 7(1):136, 2021.
[23] Geonho Hwang. Jaewoong Choi, Hyunsoo Cho, and Myungjoo Kang. MAGANet: Achieving combinatorial generalization by modeling a group action. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 44237-14288 PMLR, 23-29 Jul 2023. URL MttK:ttts:/pprceeddngg mlr www...................... wwtpps/e/e///aaeeeeeeeeeeee w2t////w22////wbbbb//// hhmmllmmm.lmmm
[24] Anubhav Jain, Shyue Ping Ong, Geoffroy Hautier, Wei Chen, William Davidson Richards, Stephen Dacek, Shreyas Cholia, Dan Gunter, David Skinner, Gerbrand Ceder, et al. Commentary: The materials project: A materials genome approach to accelerating materials innovation. APL materials, 1(1):01 1002, 2013.
[25] Yiding Jiang, Shixiang Shane Gu, Kevin P Murphy, and Chelsea Finn. Language an abstraction for hicrarchical deep reinforcement learning. Advances in Neural Information Processing Systems, 32, 2019.
[26] Chaitanya K. Joshi, Cristian Bodnar, Simon V. Mathis, Taco Cohen, and Pictro Liò. On the expressive power of geometric graph neural networks, 2023.
[27] Peter Bjørn Jørgensen and Arghya Bhowmik. Deepdft: Neural message passing network for accurate charge density prediction, 2020.
[28] Ask Hjorth Larsen, Jens Jørgen Mortensen, Jakob Blomqvist, Ivano Castelli, Rune Christensen, Marein Dulak, Jesper Friis, Michael N Groves, Bjørk Hammer, Cory Hargus, Eric Hermes, Paul C Jennings, Peter Bjerre Jensen, James Kermode, John R Kitchin, Esben Lconhard Kolsbjerg, Joseph Kubal, Kristen Kaasbjerg, Steen Lysgaard, Jón Bergmann Maronsson, Tristan Maxson, Thomas Olsen, Lars Pastewka, Andrew Peterson, Carsten Rostgaard, Jakob Schiøtz, Ole Schütt, Mikkel Strange, Kristian S Thygesen, Tejs Vegge, Lasse Vilhelmsen, Michael Walter. Zhenhua Zeng, and Karsten W Jacobsen. The atomic simulation environment-4 python library for working with atoms. Journal of Physics: Condensed Matter, 29(27):273002, 2017. URL http:// hattt::/stacks.. stacks. lop. www. 0953009533303 /984/29/ 2 273022
[29] Lin Lin and Jianfeng Lu. A mathematical introduction to electronic structure theory. SIAM, 2019.
12

---

[30] Lin Lin, Jianfeng Lu, and Lexing Ying. Numerical methods for Kohn-Sham density fune-
tional theory. Acta Numerica, 28:405-539. May 2019. ISSN 0962-4929. 1474-0508. doi: 10.101 00.1017750992449991900004
URL htttps://ww.camtt wwwbrrigaabbde.....mm crg//oor//jouraall......
acta caae-numerin /a/ artiicleeeeeaaaa..
http:////aaaaeeeee//eeer..o..r.e numerical wew.raethoesssseeeeeeeeeeeeoo - for kohnsham density----..... hetaasshiiineettttttttttttoooooo funcciionanll... thenrattoyyctitooootteeoooo/o theory/
(550FB83349DD5FIBBII 1811116011111111111111 Publisher: Cambridge University Press.
[31] Richard M Martin, Electronic structure: basic theory and practical methods. Cambridge university press, 2020.
[32] Eric J. Michaud, Ziming Liu, and Max Tegmark. Precision machine learning. Entropy, 25(1), 2023. ISSN 1099-4300. doi: 10.3390/c25010175. https:/:://// mdpi com/ /0tp//0999994////////
[33] Milton L Montero, Jeffiey S Bowers, Rui Ponte Costa, Casimir JH Ludwig, and Gaurav Malhotra. Lost in latent space: Disentangled models and the challenge of combinatorial generalisation. arXiv preprint arXiv:2204.00 022283, 2022.
[34] Milton Llera Montero, Casimir JH Ludwig, Rui Ponte Costa, Gaurav Malhotra, and Jeffrey Bowers. The role of disentanglement in generalisation. In International Conference on Learning Representations. 2021. URL httppp////pppeeeeiiieee wttt://oruum.dddoiit/131111
[35] Shyue Ping Ong, William Davidson Richards, Anubhav Jain, Geoffroy Hautier, Michael Kocher, Shreyas Cholia, Dan Gunter, Vincent L Chevrier, Kristin A Persson, and Gerbrand Ceder. Python materials genomies (pymatgen): A robust, open-source python library for materials analysis. Computational Materials Science, 68:31 08:314-315 319, 2013.
[36] John Perdew, Kieron Burke, and Matthias Emzerhof. Generalized gradient approximation made simple. Phys. Rev. Lett. 77:3665-3868, Oct 1996. doi: 10. N0.103/PhysReel.ee..7...88 URL. https //link.ape org. hwogg/dii/ia 1ttp://.1111111/11111
7865
[37] Giovanni Pizzi, Andrea Cepellotti, Riccardo Sabatini, Nicola Marzari, and Boris Kozinsky. AiiDA: automated interactive infrastructure and database for computational science. Computational Materials Science, 111:218-230 230, January 2016. ISSN 0927-0256. doi: 1o0.10//...ommmmts222200001.. attps://wwerr http://wwwwciicccirrrrtttt.t ctw http:////accoerrrrra////////////
URL
[38] Gianluca Prandini, Antimo Marrazzo, Ivano E Castelli, Nicolas Mounet, and Nicola Marzari. Precision and efficiency in solid-state pseudopotential calculations. npj Computational Materials, 4(1):72, 2018.
[39] Joshua A. Rackers, Lueas Tecot, Mario Geiger, and Tess Smidt. Cracking the quantum scaling limit with machine learned electron densities, 2022.
[40] Yousef Saad. Numerical methods for large eigenvalue problems: revised edition. SIAM, 2011.
[41] Yousef Saad, James R. Chelikowsky, and Suzanne M. Shontz. Numerical Methods for Electronic Structure Calculations of Materials, SIAM Review, 52(1):3 54, January 2010. ISSN 0036-1445, 1095-7200. doi: 10. 1137/060666633. URL httpp://epubs s am. org//oo/ (000o/111..0 1137 11113/76666666567
[42] K. T. Schüit, H. F. Sauceda, P.-J. Kindermans, A. Tkatchenko, and K.-R. Müller. SchNet - A deep lcarning architecture for molecules and materials. The Journal of Chemical Physics, 148(24):241722, June 2018. ISSN 0021-9606. doi: 10. 00.1003/1.5019777 (tttp:ttts://app http://pp.//pppssaaaaatto..oooop 1.5019779 Publisher: American Institute Physics.
scitationn
seciiaatio..og//////11... wwt//oo///....o.oooooo 1063/
[43] K. T. Schütt, M. Gastegger, A. Tkatchenko, K. R. Müller, and J. Maurer. Unifying machine learning and quantum chemistry deep neural network for molecular wavefunctions, 2019.
[44] Jimmy-Xuse Xuan Shen. mp-pyrho, Octoher 2022. URL htRpattpa:// //doi. http://000211//eeedeee 7227448 you use this software, please cite using the metadata from this file.
[45] Jimmy-Xuun Xuan Shen, Jason M. Munro, Matthew K. Horton, Patrick Huck, Shyam Dwaraknath, and Kristin A. Persson. A pnarrssentaaionnnnndeeeeet electronic charge density database for crystalline materials, 2021.
[46] Muhammed Shuaibi, Adeesh Kolluru, Abhishek Das, Aditya Grover, Anuroop Sriram, Zachary Ulissi, and C. Lawrenee Zimick. Rotation Invariant Graph Neural Networks using Spin Convolutions. arXiv:2106.0 09575 [cs], June 2021. http:tp:://aaxv.oor/abasaa
09575,
arXiv: 2106.09575.
[47] Tristan Thrush. Ryan Jiang. Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio- lingussic compositionality. 2022.
13

---

[48] Richard Tran, Janice Lan. Muhammed Shuaibi. Brandon M. Wood, Siddharth Goyal, Abhishek Das, Javier Hleras-Domingo, Adeesh Kolluru, Ammar Rizvi, Nima Shoghi. Anuroop Sriram, Fć lix Therrien, Jehad Abed, Oleksandr Voznyy. Edward Sargent, Zachary Ulissi, and C. Lawrence Zitnick. The open catalyst 2022 (OC22) dataset and challenges for oxide electrocatalysts. ACS Catalysis, 13(5):3066 3084, feb 2023. doi: 10.1021/acscatee
URL btttsss/:///do org/ 10. 1021X2Facscatal 2c05426
[49] Martin Uhrin, Sebastiaan P Huber, Jusong Yu, Nicola Marzari, and Giovanni Pizzi. Workflows in aiida: Engineering a high-throughput, event-based engine for robust and modular computational workflows. Computational Materials Science, 187:1 10086, 2021.
[50] Marin Vlastelica, Michal Rolinck, and Georg Martius. Neuro Neuro-algorithme algorithmic policies enable fast combinatorial generalization. In Marina Meila and Tong Zhang, editors, Proceedings the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 05575-00855. PMI.R, 18-24 Jul 2021. URL htt.:tt::p//oceeeeiiiiiee
mlr wtt.......eeee///////aaaaaaaaaae
html
[51] Tian Xie and Jeffrey C. Grossman. Crystal graph convolutional neural networks for an accurate and interpretable prediction material properties. Phys. Rev. Lett. 120 Apr 2018. doi: 10. 103/ PhysRevI.ett. 120. 145301. URL https://k//: org. httgg/do/110.// wwt.:o11///hhhhhhhhheee
120. 145301
[52] Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it?, 2023.
[53] Leonardo Zepeda-Núñezz Yixiao Chen, Jiefu Zhang, Weile Jia, Linfeng Zhang, and Lin Lin. Deep Density: Circumventing the Kohn-Sham equations via symmetry preserving neural networks. Journal of Computational Physics, 443:1 10523, Octoher 2021. ISSN 0021-9991. doi: 10. 202. 10522..
[54] C. Lawrence Zitnick, Lowik Chamussot, Abhishek Das. Siddharth Goyal, Javier Heras-Dominger Caleb Ho, Weihua Hu, Thibaut Lavril, Aini Palizhati, Morgane Riviere, Muhammed Shuaibi, Anuroop Sriram, Kevin Tran, Brandon Wond, Junwoong Yoon, Devi Parikh, and Zachary Ulissi. Introduction to Electrocatalyst Design using Machine Learning for Renewable Energy Storage. arXiv:2010.0940 09435 [cond-mat], October arvi:: 2010.09435.
2020. URL aattp://arxiv org/ ab1//010 2010. 094435
[55] C. Lawrence Zitnick, Abhishek Das, Adeesh Kolluru, Janice Lan, Muhammed Shuaibi, Anuroop Sriram, Zachary Ward Ulissi, and Brandon M Wood. Spherical channels for modeling atomic interactions. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. htllatpps//ppaeeeieeeeeeeeeeeeee
523GURcQQ
14


Reviews:
[{"review_id": "VzbB0RE7me", "reviewer": "Reviewer_iUWH", "summary": "The authors build and train approximately E(3) equivariant graph neural networks based on Spherical Channel Network to learn the Kohn-Sham Charge-Density scalar field as a means of accelerating density functional theory (DFT) calculations by shortening the number of self-consistent field calculations needed to achieve a converged density. The authors are motivated by finding a generalizable approach to evaluate the energetics of the combinatorial space of binary, ternary, and beyond heterogeneous catalytic surfaces, which the authors point out is poorly predicted by current pure ML approaches. The model uses a virtual node to sample the value of the charge density at specific locations that are compared to ground truth DFT calculations computed at the level of PBE using the open-source package Quantum Espresso. \n\n", "questions": "Below I give concrete feedback on specific parts of the paper section by section.\n\n> Introduction\nVery clear and well written. A concise and thorough description of the problem statement and goals of the paper.\n\n> Related Work\nAnother reference that could be included in related work is the following, which explicitly discusses using ML predicted charge densities of molecules as starting points for calculations:\nhttps://www.nature.com/articles/s41467-019-12875-2\nIn that paper they found minimal advantage to using ML charge densities but they only explored this for a small number of molecules (possibly only 1). Do your results align or disagree with their findings?\n\n> Methods\n>> 3.1\nI would be helpful to discuss further how the query points are selected. Additionally, it would be helpful to discuss the pros and cons of using a query point vs predicting basis functions (either plane waves or linear combination of atomic orbitals). I can appreciate that the query point is most flexible but you do seem to need a very large number of samples per structure. Quantitative comparison of the different charge density representations would be helpful.\n\n>> 3.2\nOne concern I have about using small unit cells is that this significantly limits what “patterns'' of different atomic neighborhoods the model sees and limits the significance of your results – since we don’t get a sense for how much data and samples would be needed to get comparable performance across more diverse environments (even if restricted to the same lattice). There are many ways to choose sites of a lattice but if the unit cells have lattices vectors only ~4 Å long, you’re only going to be able to explore fairly high symmetry configurations.\n\nRegarding the data selection based on oxidation state, one could consider encoding the oxidation state as an input feature in such a way that the model may be able to learn to generalize. For example, one could augment each atom type with an oxidation state feature.\n\n>> 3.3\nTypo: “this baseline is high non-trivial”\n\n> Results\nThe labels of Figures 2 and 3 are very small. Please improve the label sizes, choose an appropriate number of significant figures reported for MAE and Max Error, and specify which materials (mp-id(?) Or chemical formula) are being shown.\n\n>> 4.2\nType: “of of”\n\n>> 4.3\nHow are these predictions converted to your input charge density to Quantum Espresso? My understanding is that you need to convert it into plane waves – are there any subtleties in doing this?\n\nThe paper would be improved if the authors gave more specific metrics on the impact their results imply for performing general DFT calculations. The authors could better quantize how much compute is used for DFT calculations and how much better initial charge density predictions could save in compute. This would be a more powerful metric than just the number of examples that see any improvement in SCF steps. For example, a large fraction of the USA’s National Laboratory supercomputing resources are spent computing DFT – how much would better starting points save for a given level of accuracy? How much accuracy is needed for ML charge density prediction to be practical for these facilities to adopt into workflows?\n\nSomething that might seem like a silly benchmark but may be informative is what is the likelihood of reducing the number of SCF steps with random noise on the charge density (physically motivated noise but still noise). This would give more information as to whether what you are predicting requires learning or whether it’s due to the finicky nature of SCF calculations.\n\nAn additional sanity check that I would like to see is what is the difference between the converged charge density and the energy for the standard vs. ML accelerated calculations. It’s important to check whether the introduction of an ML derived initial charge density introduces any bias in the calculation results. It may very well be that there is no difference – but it’s important to check!\n\n> Broader Impact\nHere’s another great place to discuss how many hundreds of millions of hours of compute are spent on DFT by researchers and what potential savings good charge density starting points might enable.\n", "limitations": "Yes, the authors are very clear about the limitations of their methods and what can be improved on. They are also clear about what are open questions in the field (section 2.1) and what is and is not known. The authors come off as clear, knowledgeable, and thoughtful.\n\nThe authors are aiming to accelerate DFT calculations which are already used used broadly. They specifically address the potential concerns of making these insights more readily available. ", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The paper is extremely clear and well written, especially considering the need to present internal details of DFT to a general ML audience. The authors emphasize that combinatorial generalization, the ability of a model to generalize to catalytic surfaces that have combinatorial many possible substitution of constituent atoms, is a necessary metric to satisfy if ML will have impact in the heterogeneous catalysis space.", "weaknesses": "The largest weaknesses of the paper are that the charge density predictions are made on a relatively few and specific set of structures (it is unclear how well this approach will generalize to more diverse structures and atom type distributions) and that results are not presented in a way makes it easy to assess the impact of the results on the broader ML+DFT community.\n\nFor this reason, I give a “weak reject” but very much hope the authors will continue to pursue this line of work as I believe it to be a powerful approach to aiding materials design. I would also consider changing my rating if the authors can quantify the benefits of their approach more strongly. I give more details below in the Questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful review. \n\n(1)\n\nWe are happy to add a discussion of Schütt et al. (2019) to our related work section. In short we think our results are complementary. The primary differences between our works is that (1) we study learning of the charge density for bulk catalyst systems using the SCN model with an explicit focus on combinatorial generalization (2) they study learning elements of the Hamiltonian matrix directly for single molecules (water, malonaldehyde, uracil) using the SchnetOrb model (a modified version of Schnet, an early predecessor to SCN). They show SCF savings for malonaldehype (Fig 4).\n\n(2)\nWe select query points (QP) from the (real-space) grid associated to each structure. We take a maximal approach and assign a query point to  *all*  points in the grid. The possibility of more efficient selection method, e.g. through sampling, is an interesting one. In early experiments we tried some sampling approaches, but found this sometimes lead to prediction errors in poorly sampled regions, which we found detrimental to SCF convergence. We leave further investigation of this for future work. We note that this and other optimizations do not change the main message of our paper, which is evidence for combinatorial generalization in catalyst systems.\n\nRegarding a QP approach vs predicting basis functions, we choose the former because it naturally fits into the existing framework of geometric graph learning approaches, e.g. the QP is simply another geometric node in the graph and thus any equivariant properties of the model are not violated. We had considered learning the density in reciprocal space, but it is not clear to us how to define graphs there.\n\n(3)\nWe choose to limit cell sizes for reasons of computational burden, in particular to help constrain the maximum grid size per structure. For example, one large grid we found was ~10M points, which would occupy a substantial fraction of our dataset and therefore our training budget. To emphasize more diversity in structures, in the sense of different combinations, we imposed this constraint.\n\nAdditionally, for the new larger dataset, we relaxed these constraints to (1) max 12 atoms per cell (2) maximum volume of 200 A^3, and see improved results. Generalization to yet larger and more diverse structures still remains to be demonstrated, and is an interesting point of continuation for future work. \n\n(4)\n\nThanks for the suggestion about encoding oxidation states as a feature. One tricky part about this is how to do it in such a way to that preserves equivariance. One approach we tried was to encode oxidation states using the atomic index (i.e. every unique oxidation state gets a unique index). But we found this to not improve results. We leave this for future work.\n\n(5)\n\nWe appreciate the reviewer's concern about presenting our results in a way to better assess impact on the ML+DFT community.\n\nWe now include a measure of the relative number of SCF iterations saved at convergence. See Table 2 and Figure 2 in the attached.\n\nWe agree this work may potentially have large impact through saving costs, e.g. at national labs. Thanks for bringing that point up. We think our work is a step in that direction.\n\nWe weren't able to perform the \"physically motivated noise\" benchmark you proposed, but generally our experience is the SCF cycle is very sensitive to noise. Let us know in the discussion phase if this is still a concern for you.\n\nThanks for bringing up the check energy bias. We checked this and found virtually no difference between the energies. See the general comments for details.\n\n(6)\n\nRegarding converting predictions to input charge density for Quantum Espresso, we give some details on this in the Supplementary. In short all that is required is to match the file format of charge density HDF5 files which QE natively supports. We did find some sensitivity regarding the order in which data is written, but we were able to verify our implementation on ground-truth cases.\n"}, {"review_id": "Rnq3VRSEk9", "reviewer": "Reviewer_Fg4b", "summary": "- The authors investigate charge density modeling in catalyst systems. They generated a dataset using Quantum Espresso on Materials Project, defined a metric of SCF convergence savings, and tested models, focusing on generalization and comparing it to ACP initialization. They used GNNs with virtual atoms as query points. With reduced size SCN, 45% of binary cases and 63% of ternary cases achieved positive savings.", "questions": "- How does performance compare between 1. exact materials (stoichiometry + crystal structure) in train, 2. elemental combinations that were seen in train (but not the same material), and 3. elements that were seen in train but combinations that were not? (Assuming you can't test on individual elements that were not seen in train). I am curious how high performance you can even get on in-distribution materials, and whether the low generalization performance is due to lack of generalization of just underfitting.\n\n- How granular is your grid? How granular does it need to be to be useful?\n- Is it possible to make it faster but wrong? i.e. how does one make sure that the converged result is still correct?\n", "limitations": "- The authors acknowledge several limitations as potential future work: larger of systems in dataset, larger models, catalyst systems with adsorbates in addition to bulks alone, understanding why binaries are worse than ternaries.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This tackles an important problem and is a good complement to all the models directly predicting properties like energies and forces\n- The new dataset will be useful in this field\n- The paper is clearly written and easy to follow", "weaknesses": "- Only 50% of samples achieving positive savings, and binaries doing worse, do not seem like great results or something usable in practice.\n- More comprehensive analysis would be nice, e.g. why do errors occur near core regions of atoms (and is that good or bad?)\n- This paper only tests one model, which is based off an existing one, limiting the scope of contribution. Could there be different architectures better suited for charge density prediction?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review. \n\n(0)\n\nWe did try other model architectures such as Schnet, Dimenet++, and Gemnet, but we found the SCN to be the best performing. Our contribution is not specifically about finding the best architecture for density prediction, rather it is about investigating a specific kind of generalization, which we believe is interesting in its own right. However we are happy to include results with other models in the next revision.\n\n(1)\n\nRegarding underfitting, we find performance on the training set is generally better than on the test set, e.g. see MAE and savings results for the train split in Figures 1,2 and 3 in the attached. Our model was trained for 420k iterations (batchsize=44). It's possible that further training would help. But our experience tells us that adding more data may be more important.\n\nIt is true in this work that we do not address variation that occurs *within* a combination (i.e. item 2 from question 1). This kind of variation is certainly important, but it is not the focus of our work.\n\n\n(2)\n\nGrid sizes vary by structure. For instance, an example of a small grid is mp-129 with size 19x19x19 (= 6859) and a large one is mp-23329 with 127x127x127 (= 2048383).\n\nIn plane-wave DFT, grid size is determined by primarily by the the energy cutoffs and cell size. We use the energy cutoffs recommended by the relaxation workflows from [1,2], which are the result of a rigorous study of pseudopotentials.\n\n(3)\n\nThis is a good question. We checked for differences between the energies from converged ML and ground-truth densities and found no bias. Empirically this supports the claim that ML densities are converging to the \"correct\" solution.\n\n* [1] https://www.nature.com/articles/s41524-021-00594-6\n* [2] https://www.nature.com/articles/s41524-018-0127-2"}, {"review_id": "sEjshf23EL", "reviewer": "Reviewer_XGHX", "summary": "In this work, the researchers created a novel dataset of bulk catalyst relaxations with charge densities using open-source software. They aimed to investigate the phenomenon of combinatorial generalization, which refers to the ability of a model to effectively perform on new combinations of elements that were not seen during training. To evaluate the performance of their model, the researchers proposed a metric based on the comparison of convergence rates between their approach and standard charge-density initialization baselines. This metric served as an indicator of the quality of the results obtained. The experimental results demonstrated evidence of combinatorial generalization, indicating that the model performed well on new combinations of elements that were unseen during training. The proposed metric showed that the learned charge densities produced better results than the standard baseline in over 50% of the cases.", "questions": "* What is the loss used for the proposed method?\n* Are the results reproducible? \n* Can the new proposed dataset be used by different approaches or is specific to the proposed method?", "limitations": "While the paper presents a novel approach and introduces a new dataset, one limitation is the absence of the dataset itself and the code required to replicate the results and gain a deeper understanding of the proposed method. The availability of the dataset and code is crucial for reproducibility, as it allows us to verify the findings. Therefore, it is essential to consider releasing the dataset and code to facilitate transparency and reproducibility.", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The strengths of this paper are:\n\n* Contribution done in the field of catalysis.\n* Compute a new medium-scale DFT dataset of relaxed bulk catalyst structures with charge densities.\n* Proposal of a new metric used to verify the quality of results.", "weaknesses": "* The reference to figures is confusing. In lines 195 and 196 you write \" ...Figures 4.2 and  4.2 respectively.\". Same in lines 210 and 211. \n* There is no discussion regarding Figures 4 and 5 and the results they represent. \n* Although the code is not provided there is no high-level explanation of the architecture of the model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review. We realize we didn't make a number of details clear in the main text. We will make sure to add these in the next revision.\n\n(1)\n\nThe model is trained with an regression loss. Following other work in ML for catalysis, final prediction losses are reported as \"mean absolute error\" (MAE) [0].\n\n(2)\n\nWe agree with the reviewer on the importance of reproducibility and fully intend to release all code and data upon publication. We realize we didn't make this clear in the main text, only the Supplementary. We also describe our code and hyperparams there, which should ensure full reproducibility of our results.\n\n(3)\n\nWe created our dataset to specifically study combinatorial generalization in catalyst systems. However our experience tells us that researchers often find new and creative ways to use datasets that go beyond the original design. For instance, as another reviewer pointed out, other learning targets such as planewaves or atomic orbitals are alternative approaches. These would be possible with our dataset.\n\n[0] https://arxiv.org/abs/2206.14331"}, {"review_id": "gRp4qtbk0N", "reviewer": "Reviewer_7uBr", "summary": "The paper proposes a new learning framework for using machine learning to approximate chemical simulations. While many papers focus on direct property prediction (e.g. energy, forces) when applying machine learning to chemical simulation, this paper motivates the prediction of charge-density which holds promise for generalizable models across different materials systems. Following the introduction and motivation of the problem, the paper then describes related work with relevant methods in equivariant graph neural networks, materials property prediction and model generalization. Next, the authors describe their method for generating a charge-density based DFT dataset using an open-source computational chemistry engine and provide relevant statistics for their dataset, as well as a new metric (s-AUC) to measure modeling performance. The authors then describe the results of training a modestly sized SCN network on their dataset and provide an analysis of the SCN model's performance as part of the charge density prediction workflow compared to the currently applied method in Quantum Espresso calculations. Overall, the results indicate that the SCN can in some cases similar precision to the original method while being significantly more compute efficient.", "questions": "Some questions for the authors:\n* Can you provide additional detail on the types of materials systems your SCN model performed well on in addition to the binary and ternary classification? It could be interesting to look at generalization from the materials construction perspective.\n* Can you provide additional details for the potential compute benefits of using SCN for charge density prediction?\n* The high-precision learning motivation you have in Section 2.4 is quite relevant to this paper. Could you motivate this further in terms of high precision is needed. This might tie into applying these models in the computational workflow that you briefly mention, which would also be nice to expand upon.", "limitations": "The authors provide a discussion of limitations in their paper, which list a set of relevant points. It would be nice to include a discussion on other types of open-source solvers, i.e. not Quantum Espresso, as well.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper has the following strengths:\n* Originality: The paper proposes a new learning target for computational chemistry simulations, charge-density, that is motivated by a deep understanding of the physics and chemistry of the Kohn-Sham equations. Based on authors' review, there is few prior works addressing this question even though proper models could lead to greater generalizability.\n* Quality: The authors define a new performance metric (s-AUC), define a new dataset to ensure consistency and measure modeling error rates to a high precision.\n* Clarity: The performance metrics are generally well defined and the plots in Figure 4 and Figure 5 provide a good overview of the SCN model's performance.\n* Significance: Properly training charge-density models can be significant for the application of machine learning models to computational chemistry. The problem statement has good originality and significance.", "weaknesses": "The paper could be improved by:\n* Providing greater summary statistics of the SCN model's performance across the entire dataset studied. While Figure 4 and Figure 5 are useful, it is currently hard to understand how well SCN performs across the entire dataset. Table 2 provides some statistics, but the Figure caption is not informative in trying to understand the content of the Table. [Clarity, Quality]\n* It would be nice to see how effective charge-density calculation can lead to the prediction of properties found in other studies. The authors mention some reference in the text, but it would be nice to see some of those demonstrated in the paper and thereby showcase how the two methods compare. [Significance]\n* The authors mention the release of related code and data in the supplementary material. Given that the dataset and workflow are the primary contributions of this work, it would be nice to see that clarified in the main paper. [Clarity]\n* It would have been nice to see the evaluation of additional models on the generated dataset or at least an explanation as to why they were not considered. [Quality]\n* Additional data related to compute efficiency increases of the proposed method would help make a stronger case for the potential benefits of the method. [Significance]", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review. \n\nRegarding the performance of SCN on the entire dataset, please see the distributions of metrics in Figures 1 and 2. Most scores fall into this range of -0.5 and 3 in Figure 1, and between -0.2 and 0.8 in Figure 2.\n\nRegarding other models, we did try several other models like  Schnet, Dimenet++, and Gemnet in our preliminary experiments, but found SCN to perform the best. We are happy to include these other models in the next revision if it is a concern to you.\n\nWe will certainly clarify our intent to release all code and data in the main text.\n\n(0)\n\nRegarding computing other properties from the charge density, in addition to the energy bias check, we also compared the band gaps computed from the ground-truth and ML densities. We do so using the built-in tool in ASE [2]. We find 95% of values to be exactly the same, with maximum relative error at $7.8\\times 10^{-5}$. \n\n(1)\n\nAnalyzing top performers for patterns is an interesting idea and could useful for improving the model or dataset. We attempted such an analysis from a view different angles, but didn't find anything conclusive to report. This is an interesting question for future work.\n\n\n(2)\n\nYes, we have now included percentage of iterations saved. See Table 2 and Figure 2.\n\n(3)\n\nWe are glad the reviewer found the section on high-precision relevant. We included it because the precision requirements for DFT applications are often quite high, e.g. the convergence threshold we used was 1e-10.\n\nWe want to point out the distinction between high-precision in training and testing. We think the later, concerning generalization to new structures with high precision, is a particularly challenging. \n\nAlthough our work doesn't contribute any specific training techniques as in [0], we are able to achieve high-precision by interfacing with the SCF cycle in a natural way: through initialization with learned densities. This \"hybrid approach\" is echoed in a number of places, e.g. [1], and we believe is an increasingly important alternative to approaches purely with ML.\n\nWe will add this discussion in the next revision.\n\n\n* [0] https://arxiv.org/abs/2210.13447\n* [1] https://www.nature.com/articles/s41467-019-12875-2\n* [2] https://wiki.fysik.dtu.dk/ase/ase/dft/bandgap.html\n\n"}]

Author rebuttal:
We thank the reviewers for their feedback. We address common concerns of reviewers below:

* (1) strength of performance and number of structures
* (2) assessing impact of results on ML+DFT community
* (3) comprehensiveness of evaluation
* (4) bias of learned densities

---

(1)

To address concerns about performance, we have scaled up our dataset by a factor of 8.5x more structures. We now train on 47 unary and 392 binary catalysts, spanning 73M samples of (graph, density) pairs.  We substantially increase the test set, now evaluating on 360 binaries and 1116 ternaries unseen at train time, spanning over 450M samples. We follow the same data generation procedure as described in the main text, except for relaxing constraints on the cell size (now 200 Ang^3) and number of atoms (now 12). All told, our dataset now spans 1915 structures and 525M samples.

Performance on this new data is much improved. In terms of the s-AUC metric, we improve upon the baseline in 83% and 86% of binary and ternary test cases respectively. We summarize these new results in Table 1.

(2)

We care about presenting our results in a way that makes it easier to assess impact on the ML+DFT community. To that end, we have also computed savings in terms of the percentage of SCF iterations saved at convergence. We think this should be more interpretable to the community. We find that we save on average of 13% of iterations in both binary and ternary cases. We report summary statistics for iteration savings  in Table 2.

To put it as suggested by one reviewer, supposing that the USA National Labs spend \\$100M on DFT per year and negligible cost of inference, then a 13% reduction in SCF iterations would yield a savings of \\$13M. There are of course a number of practical details to work out to achieve this, but we believe this highlights the potential impact of our work.

(3)

We care about providing the full picture of our results. To that end, we show the distributions over structures of s-AUC scores and iterations saved in Figures 1 and 2. These plots and the summary statistics give the complete picture.

(4)

Some reviewers raised the question of bias, i.e. whether learned densities might converge to a different solution than the ground truth. To check this, we compared the energies at convergence between the ground-truth and learned cases. We find almost zero difference between these energy values. On the test set we found 88% of structures had exactly zero difference in energy values. Of the cases with non-zero error, the relative error on average was $1.2\times 10^{-11}$ and at a maximum of $1.5 \times 10^{-9}$. Empirically it appears there is very little difference.


Meta-review: This paper tackles the problem of charge density modeling in catalyst systems, an important one in the area of ML + Materials Science.  On the one hand, there was unanimity among reviewers that the paper tackles an important problem, that it is presented clearly and accessibly to a ML audience, and that it is likely to be of significance to the community. In addition, the reviewers appreciated the usefulness of the dataset introduced in this paper. On the other hand, there were concerns about the scope of the experimental validation (predictions made on a narrow set of structures, only a single model tested, etc). The rebuttal by the authors, however, was thorough and addressed many of these concerns. Overall, despite its limitations, this paper is likely to be of interest to the ML + Density Functional Theory (DFT) community.
Decision: Accept (poster)