Infusing Self-Consistency into Density Functional Theory Hamiltonian Prediction via Deep Equilibrium Models
Authors: Zun Wang
Chang Liu
Nianlong Zou
He Zhang
Xinran Wei
Lin Huang
Lijun Wu
Bin Shao
Venue: NeurIPS 2024 poster (2024)
Keywords: electronic structure
deep equilibrium model
Primary area: machine_learning_for_other_sciences_and_fields

Abstract: In this study, we introduce a unified neural network architecture, the Deep Equilibrium Density Functional Theory Hamiltonian (DEQH) model, which incorporates Deep Equilibrium Models (DEQs) for predicting Density Functional Theory (DFT) Hamiltonians. The DEQH model inherently captures the self-consistency nature of Hamiltonian, a critical aspect often overlooked by traditional machine learning approaches for Hamiltonian prediction. By employing DEQ within our model architecture, we circumvent the need for DFT calculations during the training phase to introduce the Hamiltonian's self-consistency, thus addressing computational bottlenecks associated with large or complex systems. We propose a versatile framework that combines DEQ with off-the-shelf machine learning models for predicting Hamiltonians. When benchmarked on the MD17 and QH9 datasets, DEQHNet, an instantiation of the DEQH framework, has demonstrated a significant improvement in prediction accuracy. Beyond a predictor, the DEQH model is a Hamiltonian solver, in the sense that it uses the fixed-point solving capability of the deep equilibrium model to iteratively solve for the Hamiltonian. Ablation studies of DEQHNet further elucidate the network's effectiveness, offering insights into the potential of DEQ-integrated networks for Hamiltonian learning. We open source our implementation at https://github.com/Zun-Wang/DEQHNet.

Infusing Self-Consiseency into Density Functional Theory Hamiltonian Prediction via Deep Equilibrium Models
Zun Wang Mierosoft Research A14Science Beijing, China, 100084
Chang Liu Microsoft Research A14Science Beijing, China, 100084
zunwang@miiroosof..co com
Nianlong Zou
State Key Laboratory of Low Dimensional Quantum Physics and Department of Physics, Tsinghua University, Beijing, 100084, China
He Zhang*
Xinran Wei
Xi' an Jiaotong University, Xi'an, China
Microsoft Research AI4Science Beijing, China, 100084
Lin Huang
Lijun Wu
Microsoft Research AI4Science Beijing, China, 100084
Microsoft Research A14Science Beijing, China, 100084
Bin Shao
Microsoft Research A14Science Beijing, China, 100084 binshao@micrr www.:e/mmmmoooooo.ooooooooooo com
Abstract
In this study, we introduce a unified neural network architecture, the Deep Equilib- rium Density Functional Theory Hamiltonian (DEQH) model, which incorporates Deep Equilibrium Models (DEQs) for predicting Density Functional Theory (DFT) Hamiltonians. The DEQH model inherently captures the self-consistency nature Hamiltonian, a critical aspect often overlooked by traditional machine learning approaches for Hamiltonian prediction. By employing DEQ within our model architecture, we circumvent the need for DFT calculations during the training phase introduce the Hamiltonian's 's self-consistency. thus addressing computational bottlenecks associated with large or complex systems. We propose a versatile framework that combines DEQ with off-the-shelf machine learning models for predicting Hamiltonians. When benchmarked on the MD17 and QH9 datasets, DEQHNet, instantiation of the DEQH framework, has demonstrated signifi- cant improvement in prediction accuracy. Beyond a predictor, the DEQH model is Hamiltonian solver, in the sense that it uses the fixed-point solving capability of the deep equilibrium model to iteratively solve for the Hamiltonian. Ablation studies of DEOHNet further elucidate the network's effectiveness, offering insights
'These authors did this work during an internship at Microsoft Research AI4Science.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

into the potential of DEQ-intteraateee networks for Hamiltonian learning. We open source our implementation httpps: gi github. wttp.hcooooooooocoooooooo.oo.oot com Zun -angg www.eWanngggeeeeeeee DEQHNet
Introduction
Density Functional Theory (DFT) is a framework that has revolutionized the study of physical, chemical, and material systems. By providing insights into the electronic structure of matter, DFT has become an indispensable tool for researchers in various scientific fields. Despite its widespread adoption, the computational intensity of DFT poses significant bottleneck, particularly when applied to large or complex systems.
To navigate the computational challenges of DFT, machine learning (ML) has emerged as powerful ally, capable predicting a myriad of molecular and material properties with reduced computational
overhead [ 2
3
5
7
8
9 10
12 13 14 15, 16
17 18 19 20 21. However, these
ML approaches typically require bespoke models for different properties, necessitating extensive
retraining redesign.
The Hamiltonian, a fundamental quantity in DFT, encapsulates the entire energy and dynamics of a system. Properties of molecules and materials can be considered downstream tasks of the Hamiltonian. Therefore, once the Hamiltonian is accurately predicted, broad spectrum of properties can be subsequently derived. This realization has spurred interest in using ML to predict the Hamiltonian directly [22]. Recent efforts in this direction have seen substantial improvements by incorporating equivariant graph neural networks, which respect the underlying symmetries of physical
laws 23 25 26).
In the context of DFT, the Hamiltonian is not just another property- it inherently involves self- consistent iterative processes to achieve convergence. Some studies 27 28 29 30 have incorporated this iterative nature into network training by intertwining DFT computations with the loss function during the training of neural networks, and even leveraging unlabeled data to enhance the model's generalizability. Nonetheless, these methods unavoidably rely on DFT iterations to solve the general- ized eigenvalue problem, rendering them impractical for larger systems. This is because they integrate DFT iterations within the training process, melding DFT with the training loss. To fundamentally address this issue, a paradigm shift network architecture is required.
In this work, we introduce DEQH model, combines deep equilibrium models (DEQs) [31] with off-the-shelf neural network models for the prediction of DFT Hamiltonians. DEQs, characterized by their ability to model systems at an equilibrium state, offer a structural innovation that inherently captures the self-consistent nature of the Hamiltonian without the need for iterative DFT calculations during training. By embedding the equilibrium concept into the architecture, our model seeks to directly learn the fixed-point representation of the Hamiltonian, circumventing the computational complexity while maintaining fidelity to the principles of DFT. Our key contributions are:
The Hamiltonian's 's iterative qualities are often neglected by standard machine learning approaches for its direct prediction. Our approach integrates DEQs with off-the-shelf ML frameworks, leveraging node features derived from the Hamiltonian and overlap matrix to harness these iterative aspects. Considering computational efficiency and practical applica- bility, we adopt QHNet the backbone of the architecture, leading the development of DEQHNet.
Traditional machine learning models primarily serve as Hamiltonian predictors, and while recent self-consistency integrating frameworks aim to refine training, they incur high com- putational costs. DEQH model distinguishes itself by acting fundamentally iteratively determining the Hamiltonian with the deep equilibrium model's fixed-point capabilities. This intrinsic self-consistency and computational efficiency render DEQH model scalable approach for precise eigenstate prediction without a significant increase in complexity.
solver,
We have benchmarked DEQHNet against the MD17 22 and QH9 32) datasets. demon- strating that the incorporation of Hamiltonian self ceff-coniittency can significantly enhance predictive accuracy.
We conduct an ablation study on DEQHNet, and present conjectures regarding the efficacy of networks that incorporate DEQs for the task of learning Hamiltonians.

---

This architectural leap holds the potential to unlock scalable, efficient, and accurate predictions of DFT Hamiltonians, paving the way for rapid advancements in the understanding and design of new materials and molecules.
2 Related works
To date, more and more studies have tackled the intricate challenge of employing machine learning (ML) techniques to model the wavefunction directly. The first study in this domain was conducted by Hegde and Bowen [33, who utilized kemel ridge regression to ascertain the Hamiltonian matrix. Building upon this, Schütt et al. introduced the SchNOrb neural network architecture [22]. which assembles the Hamiltonian matrix of molecules using block-wise approach based on atom-pair features. Deep Hamiltonian [24] was developed for the prediction of Hamiltonian matrices of periodic systems. PhiSNet (23) leverage the principles from a series of -quiivaaaa models to ensure that predictions maintain an exact physical consistency with the orientation of the input structures. DeepH- E3 [25] formulates the DFT Hamiltonian as a function of the structural configuration of the material. This representation inherently upholds Euclidean symmetry, maintaining this invariance seamlessly even when spin-orbit coupling effects are taken into account. QHNet [26] has been meticulously constructed to eliminate 92% of the tensor product operations to enhance the computational efficiency of the model. DeepH-2 B4 is an equivariant local-coor transformer which reduces SO(3)
convolutions to SO(2) for efficient Hamiltonian prediction.
Additionally, increasing number of studies have recognized the inherent self-consistent iterative nature of the Hamiltonian's 's eigenproperties. These works integrate DFT into the training process to enhance network performance by using this self-consistent iterative characteristic. For instance, Ref. (27) introduced a self-consistent loss that utilizes unlabeled data through amortized DFT, assisting the model training and enabling the exploration of broader chemical space improve generalization capabilities. Ref. 30 circumvent the creation of datasets by pretraining with DFT iterations and employing an implicit DFT loss, denoted as E(-), as the training loss function. Simultaneously, similar approach (28) uses a network predicting the Hamiltonian matrix as an input to DFT to obtain energies, which are then used as optimization targets. Ref. (29] has approached the problem by simultaneously predicting both the Hamiltonian matrix and the density matrix, introducing the direct inversion in the iterative subspace (DIIS) error H HD SDH as a training signal to avoid the costly self-consistent field calculations. Furthermore, this method can estimate the accuracy of the predictions, making it amenable to integrate with active learning strategies.
3 Preliminary
Equivariance Consider a mapping function L : - y that transforms inputs from space A' to outputs within space V. The function L is deemed G-equivariant if it consistently upholds the symmetry induced by a group G in its mappings. Specifically, for every element g belonging to group G, the following relation is satisfied:
(1) where x and D denote the actions of the group G on the spaces X' and Vi respectively, This condition guarantees that the function L faithfully translates the symmetry operations performed on the inputs by G into corresponding transformations in the outputs. For additional concepts relevant to this discussion, please refer to the Supplementary Material A.2 A.3
LoD
DFT Hamiltonian The primary goal of most electronic structure methodologies is to accurately solve the electronic Schrödinger equation:
Ha = Ve
where Hæ is the electronic Hamiltonian operator, which encapsulates the interactions and dynamics of electrons, Vaa represents the electronic wavefunction, and E., denotes the ground state energy. Typically, the electronic wavefunction V.r is approximated by an antisymmetric combination of molecular orbitals { (r)) (r denotes the electronic coordinates). Generally, each molecular orbital d is formulated as linear combination of centecentered basis functions {Ø, (x)):
(r) Σ (r).
3

---

This formulation leads to the matrix equation: HC = SCe
(4)
in which the Hamiltonian is represented as a matrix H. The overlap matrix S, with elements Si = [ (1) (r) is introduced to account for the con-orthonormality of the basis functions. To obtain the coefficients Ci that define the wavefunction Ver and the orbital energies E. one must solve generalized eigenvalue problem. More introduction refers to the Supplementary Material A..I
Deep equilibrium model The deep equilibrium model (DEQ), a class of innovative implicit layer models introduced by Ref. [31]. has garnered considerable attention for its remarkable performance across various large-scale vision and natural language processing tasks. These models often rival or even surpass the state of the art established by traditional explicit models, as demonstrated in subsequent studies by Bai el al. [35]. The fundamental principle underpinning this methodology is the concept of an implicit layer that converges to fixed point through an iterative process.
typical k-layer deep network h X + y is defined by a stack of layers.
21 x;
2++1 (Wzz + b,), i= 1, ... , 1,
(5)
h(x) w bk-
To allow for the continuous integration of input information throughout the depth of the model, an input injection mechanism is incorporated across the layers, which involves adding a linear transformation of the input, denoted as Ur. Consequently, this augmented model can be characterized
as
21 = K,
1 (Wiz + Ux + bi), i=1,... 1,
(6)
h(x) = W& 2 + by-
To model an infinitely deep network (A: + oo), it is observed that the layer values typically converge to fixed point, 2" = a (Wz* Ur b). The inclusion of input injection Uv in the model is critical due to the equilibrium point's independence from any initial 2 value. Otherwise, the network's output would become paradoxically invariant the input. Drawing on the insight that deep sequence model layers often converge to fixed point, the DEQ approach employs root-finding methods to identify these equilibrium points, thereby optimizing the learning process. For a comprehensive explanation of DEQ, please see the Supplementary Material A.7
4 Methods
Our DEQ-based framework, depicted in Fig I is designed Hamiltonian solver, contrasting with traditional predictors. It inputs the Hamiltonian, overlap matrix, and structural data to yield the Hamiltonian output, which is then recursively processed through DEQ find its fixed point. Theoretically, the Hamiltonian and structural information alone could suffice, as structural data can function conduit within DEQ maintain fixed point traits, with atomic orbital representations learned from the structure itself (see Sec. Sec. Nevertheless, we include the overlap matrix, given its role in DFT's iterative equation (Eq, 44. ease of calculation, and similarity in node feature construction.
4.1 Diagonal reduction
The matrix element between atom i and of single-electron operator Ô E (fr represented in the atomic orbitals (Φ) is
T Ten.je = ( ()
(7)
where μ = (n1, 11 m1) and = (nz, I2, m22 are the basis functions in the set of atomic orbitals centered at each atom, respectively. The n, I, and m are principal, azimuthal, and magnetic quantum numbers, respectively. Refer OrbNet- Equii (36), for each diagonal block of T. i... TAA, defined for an on-site atom pair (A, A), there exists set of 7-independeet coefficients Qui such that the following linear transformation v/.
h4
Σ TILY Qua
(8)

---

Diagonal Reduction
Hamiltonian Solver
Zi Xt
Diagonal Reducttio
Figure I: Schematic of the DEQH model. A Hamiltonian solver must be engineered to concurrently process structural information, the Hamiltonian, and the overlap matrix, and to output the subsequent Hamiltonian iteration for DEQ convergence. Within the Hamiltonian solver, the module handling structural information is termed injection, while the remaining components collectively referred to as filter.
is injective and satisfy equivariance. The existence of Q could be proved by Wigner-Eckart theorem. For the sake of computational feasibility, physically-motivates scheme is employed tabulate Q and produce order-1 equivariant embeddings A using on-site 3-index overlap integrals Q:
Que
= =Qmi mammmmmmm mmmmmn.aa.m
[f ( di. (r))* ma (r) (r)dr FERS
(9)
where A are the atomic orbital basis, and & are auxiliary Gaussian type basis functions. Q adheres equivariance constraints due to its relation SO(3) Clebsch-Gorde coefficients C/ mm x Jeese m, (r) (r)Y,, (r)dr. Consequently, the role of Q is to couple the indices I and l2 of the Hamiltonian matrix elements, resulting in features of order 1. Details could be found in Supplementary Material A.4 A.4 A.6
4.2 Hamiltonian solver
In this paper, we have chosen to use QHNet (26) because of its versatility and adaptability. QHNet is capable of supporting multiple molecules simultaneously. Its matrix prediction module is designed to predict matrices of varying sizes for different molecules (32, making it more flexible and robust choice for our study. As illustrated in Fig. 2 we present the comprehensive architecture of DEQHNet. To effectively incorporate the self-consistent nature of the Hamiltonian through the DEQ mechanism within the network architecture, is imperative that the network's 's filters accept the Hamiltonian as input. Additionally, structural information must be used an injection to establish dependencies at the fixed point. Consequently, this necessitates the design requirements. This solver will enable the network to iteratively refine its predictions, ensuring that the output Hamiltonian self-consistent and in line with the structural information provided, thereby harnessing the full potential of the DEQ framework in electronic structure modeling.
Hamiltonian solver tailored these
Injection As indicated in the boxed area of Fig. a), the section highlighted represents the injection component of DEQHNet. Within this component, the atomic number {2) is input as the node feature, while the relative position vectors fij are processed through Radial Basis Functions (RBF) and spherical harmonics to construct the edge features that are invariant and equivariant, respectively. This design ensures that the crucial chemical information, such the type of elements and their spatial relations, are effectively encoded into the network, allowing DEQHNet to accurately capture the nuances of molecular structures and interactions.

---

(a)
(b)
(c)
50
Tnjertion
+
Dingonal Reduaction
Dingonal Reduction
Embedding
Linear
RBP
(s)
Leaily Reil
Gafee
Transtllock
TransBlock
ConvNetl ayer
L Linear
Lincar
SellNetLayer
PaireeeLLaer
Ze
Expansion
Expansion
TramBlick
ConyNetl.ayer
(d)
(c)
X
(f)
R/S
Linear
Lincar
Linear
Lincar
Línear
Linear
Diggiaa Redactluo
Expansion ())
Expansion (ii)
SefNNelaay
PairNetLLayer
Figure 2: A schematic diagram of DEQHNet. (a) The overall architecture of DEQHNet, with the injection mechanism encapsulated within the yellow frame and the remainder designated the filter. (b) TransBlock. Integrates the Hamiltonian and overlap matrix with structural information (relative position vectors). (c) ConvNetLayer. Responsible for processing structural information. (d) Diagonal Reduction. Transforms the Hamiltonian and overlap matrix into equivariant node features. (e)-(g) Couple node features to form diagonal and off-diagonal blocks of the Hamiltonian.
Filter Our design draws inspiration from QHNet (26) and Equiformer [15] to handle structural information, as well as node features constructed from the Hamiltonian and the overlap matrix. As depicted in Fig. (d). the Hamiltonian and overlap matrix initially processed using the previously described diagonal reduction method to construct node features. is noteworthy that the initial value of the Hamiltonian provided to the Hamiltonian solver set as the identity matrix I, a deliberate H/S((z)
choice to preserve the Hermiticity of the Hamiltonian. These features are then normalized as x; before being fed into the module illustrated in Fig. (b). Within this module, the equivariant node features are combined with the equivariant relative position vectors through a tensor product, resulting in the generation of new node features,
1/S..(.) haj
(Linear(<;; #/S.0r)) + Linear(x,) (1115(00)) @ Vi,m (Fig)
aij Softmax (WLeakyReLU(h H/ H1811=0))
Vig Linear
H/S,01)) hig
(fum(Fts))
(10)
H/S
Σ
2
Aij
JEM:
As depicted in Fig. 2 (c), the filtering module within the Tensor Field Network (TFN) layer regulates the influence of messages coming from other nodes. In processing structural features, the inner product I of the (-order irreducible representation on a pair calculated as
T (Linear(x:). Linear(z!))..
(11)
Subsequently, the pairwise cosine similarity of the irreducible representations, along with the 0-order irreducible expression, are fed into a MIP to compute attention scores aij. The calculation of the message mi from node j to node is constructed through tensor product of the transformed relative

---

position vector Psy and the feature vector 2/ee-
R!lnnd y (rig) (mm((()) 8/
m
(12)
The output irreducible representation z/ is then obtained by aggregating the messages my and self-connections form the updated feature:
i Linear
+
Σ
mly
(13)
JEN,
Subsequently, the node features 8." #//S constructed from the Hamiltonian and overlap matrix are merged with the node features 7 containing structural information. This amalgamation is employed to construct the iteratively refined Hamiltonian for the solver. As shown in Fig, [(e), the block is designed to harness the tensor product between nodes separately construct on-site features and pairwise features. These are then channeled into the module depicted in Fig. (g)) where the inverse operation of the tensor product [26]. i.e. the tensor expansion operation,
la
Σ
Cla Lammallamm
w
(14)
my?
applied to construct the Hamiltonian for the next iteration step of the solver.
We have developed a complete Hamiltonian solver that scamlessly integrates with the Differentiable Equa- tion (DEQ) method for fixed-point iterations, enfore- ing the Hamiltonian's 's inherent self-consistency. This solver iteratively refines the Hamiltonian, mirroring the self-consistency principle in physics. Utilizing the DEQ framework, dynamically adjusts parameters to achieve convergence, yielding Hamiltonian that pre- cisely captures the interactions. This approach not only improves accuracy but also aligns with the iterative nature of electronic structure calculations.
(a)
R
H
(b)
R
4.3 Mathematical formulations of Hamiltonian solver and predictor
H - H
Hini - H
As illustrated in Fig. 3 the DEQH model operates as Hamiltonian solver, learning the iterative process H = f Z, R). In contrast, models like QHNet function as predictors, using the formula H fZz R), where and R represent atomic numbers and coordi- nates, respectively. The function ff typically involves a neural network, and denotes the Hamiltonian, The superscript H" indicates the converged Hamiltonian.
ltcration
Figure 3: (a) The network predicts the Hamiltonian using atomic number Z and coordinates as inputs, outputting the molecular Hamiltonian H. (b) The DEQH model also includes an input for H and iter- atively refines the Hamiltonian until reach- ing a fixed-point solution H" hence we re- fer to this block as the Hamiltonian solver.
In practice, if the dataset includes overlap matrices, these can also be incorporated into the network input, transforming them into equivariant node features akin to the Hamiltonian. Even if overlap matrices are not initially present, they can be easily calculated during data preprocessing using and R. The overlap matrix provides wealth of detailed information, enhancing the model's input. Consequently, the equations for the DEQH and the modified Hamiltonian predictor become H = f(H", Z, R. S) and H fZZ S), respectively, where S is the overlap matrix.
5 Results
We evaluated the performance of DEQHNet by applying the MD17 22 and QH9 datasets. We have further explored the convergence behavior of DEQHNet. Additionally, we conducted
7

---

an ablation study to isolate and understand the specific impacts of the DEQ mechanism and the architectural design on our method's effectiveness. For descriptions of the metrics and results pertaining to acceleration ratio, please consult the Supplementary Material A.8 A.9
5.1 MD17
MD17 dataset [2 encompasses an array of molecular properties, including structures, energies, forces, and Hamiltonians V Fock Kohn-Sham matrices- as well as overlap matrices pertaining to molecules such as water, ethanol, malondialdehyde, and uracil. All computational analyses were executed the PBE/def2-SVP level of theory 38), employing the ORCA electronic structure software [39]. Table Sableesshee shows DEQHNet's performance on the MD17 dataset against QHNet. Except for water, DEQHNet achieves lower mean absolute error (MAE) in Hamiltonian predictions for three molecules, benefit likely from more extensive training data that prevents overfitting. The predicted Hamiltonian improves orbital coefficients and the MAE for orbital energies has risen, which indicates that lower Hamiltonian MAE doesn't necessarily correlate with better orbital energy predictions.
Table I: Comparison of the MAES between QHNet and DEQHNet trained on MD17 dataset. The optimal and the second best values are highlighted in bold and underlining, respectively. should be noted that due to the design of DeepH (24). the data used is re-labeled by OpenMX [4], which may lead different results compared to other models. Furthermore, considering the high training cost of PhiSNet (ori) (23). PhiSNet (reproduce) results from the QHNet [26 article are provided for more
balanced and comprehensive comparison.
Dataset
Model
H [
6 [10-9FA) 4
& [10-2]1
SchNOrb
165.4
279.3
100.00
PhiSNet (ori)
17.59
85.53
100.00
PhiSNet (reproduce)
15.67
99.94
Water
DeepH
38.51
QHNet
10.79
33.76
99.99
DEQHNet
36.07
335.86
99.99
SchNOrb
187.4
334.4
100.00
PhiSNet (ori)
12.15
62.75
100,00
PhiSNet (reproduce)
20.09
102.04
99.81
Ethanol
DeepH
22.09
QHNet
20.91
81.03
99.99
DEQHNet
18.73
106.94
100.00
SchNOrb
191.1
400.6
99.00
PhiSNet (ori)
12.32
73.50
100.00
PhiSNet (reproduce)
21.31
100.60
99.89
Malonaldehyde
DeepH
20.10
QHNet
21.52
82.12
99.92
DEQHNet
17.97
93.79
99.90
SchNOrb
227.8
1760
90.00
PhiSNet (ori)
10.73
84.03
100.00
PhiSNet (reproduce)
18.65
143.36
99.86
Uracil
DeepH
17.27
QHNet
20.12
113.44
99.89
DEQHNet
15.07
107.49
99,89
5.2 QH9
The QH9 dataset 32 crafted using PySCF 41 split into QH9-stable and QH9-dynamic. QH9- stable contains 130,831 Hamiltonian matrices from QM9 42 43 while QH9-dynamic has 100- geometry trajectories for 999 molecules. DFT calculations employ grid level of SCF convergence with 10 tolerance, and a 3.16 x 10-5 gradient limit. The B3LYP functional and deI2SVP GTO

---

basis set are used, with DIIS aiding SCF convergence. QH9-dynamic's molecular dynamics were run 300K an NVE ensemble. The 1199-dymamic-100 100k subset features 0.12 fs step trajectories, taken every 10th step for 1,000 steps. Table displays the results of DEQHNet on the QH9 dataset, where significant reduction the MAE the Hamiltonian across both the QH9-stable and QH9-dynamic subsets can be seen. The decrease in MAE is particularly striking for the QH9-dynamic subset, where nearly halves. The fidelity of the orbital coefficients, derived from the Hamiltonians predicted by DEQHNet, reached over 99% accuracy. This is attributed to the fact that the overlap matrix and the Hamiltonian offer significant insights into the basis set, thereby facilitating the expression of orbital coefficients. Similar to the trend observed with the MD17 dataset, there is direct positive correlation between the MAE of the Hamiltonian and the orbital energies. Additional experiments was performed elucidate this situation, with details provided in the Supplementary Material (a.
Table 2: Comparison of the MAEs between QHNet and DEQHNet trained on QH9 dataset. The optimal values are highlighted in bold.
H [10 "E,] 1 non-diagonal
Dataset
Model
[10
"E,1
& [10 2]
diagonal
all
QHNet DEQHNet
111.21
73.68
76.31
798.51
95.85
QH9-stable-id
96.43
58.75
61,42
4383.10
99.84
QHNel 111.72
69.88
72.11
644.17
93.68
QH9-stable-ood
DEQHNeL
81.01
51.66
53.23
5657.07
99.80
QHNeL 149.62
92.88
96.85
834.47
94,45
QH9-dynamic-ger -eo
DEQHNet
84.97
60.04
62.14
1864.06
99.92
QHNel 416.99
153.68
173.92
9719.58
79.15
QH9-lynamic-mol
DEQHNet
210.76
97.18
105.80
4625.88
99.80
(8)
(b)
PySCF DECHNez
0.04
MAX 110
0.01
25000 50000
75000
100000
125000 150000 Steps
175000
200000
10
terations
Figure 4: (a) The variation in the number of DEQ iterations within DEQHNet as a function of training steps. For this analysis, 50 random configurations were selected from the uracil test set, and the iteration counts were tallied after performing inference with DEQHNet checkpoints saved during the training process. (b) The change in Hamiltonian MAE with respect to iteration count for randomly chosen molecule from the QH9-dynamics-geo test set, comparing the self-consistent field (SCF) iterations using PySCF, DEQHNet inference, and DEQHNet inference initialized with Hamiltonians guessed by PySCF.
5.3 On the convergence of DEQHNet
We further explored DEQHNet through additional experiments. As depicted in Fig. 4 a). we inferred 50 uracil test set configurations using DEQHNet checkpoints from various training phases. Initially, the model required many iterations to achieve convergence but soon stabilized at two or three iterations, showing quick adaptation to the Hamiltonian's 's iterative solution. comparison, PySCF calculations for these configurations, which are structurally similar due to their molecular dynamics origin, consistently converged 15 SCF iterations. Initially, DEQHNet surpassed this number but soon settled at fewer iterations. This suggests that DEQHNet learns distilled version of the DFT

---

solver, and unlike PySCF, which considers energy for convergence, it uses Hamiltonian differences for this purpose.
Furthermore, we randomly selected a molecule from the QH9-dynamic-gen test set for additional analysis. In Fig. 4b), we plotted the MAE of the Hamiltonian at each iteration during the PySCF computation and the DEQHNet inference relative to the true Hamiltonian. DEQHNet starts with an identity matrix as the initial Hamiltonian, leading to high initial error, while PySCF employs MINAO density --he superposition of orbitals of each atom in isolation-for its initial guess, resulting in lower starting MAE. DEQHNet typically stabilizes within three iterations, unlike PySCF, which can have initial optimization fluctuations. By initiating DEQHNet with PySCF's starting Hamiltonian [4]] we noted improved stability in convergence. This suggests that DEQHNet's iterative refinement can benefit from a more accurate initial guess, enhancing the stability and reliability of convergence, an important aspect in electronic structure computations.
5.4 Ablation studies
To evaluate DEQHNer's efficacy, we conducted ablation studies with two model variants: one excluding the overlap matrix from DEQHNet and another enhancing the original QH- Net with overlap matrix features. As depicted in Fig. [] DEQHNet's s per- formance dips without the overlap matrix, while the upgraded QHNet shows reduction in MAE for Hamil- tonian prediction but doesn't surpass DEQHNet. The results suggest that both models can infer atomic orbital details from molecular structures and the Hamiltonian labels. The overlap matrix, however, seems to expedite the network's s acquisition of atomic orbital information, aiding Hamil- tonian construction. Supplying the overlap matrix equips the network with explicit atomic orbital data, fa- cilitating the learning of the effec- tive potential from structural inputs. Without this data, the network must align with the Hamiltonian's 's basis while learning atomic orbital information through loss optimization.
DEQHNet DEQHNet
140
OHNet
QHNet w/S
100
MAE
80
20
0
H100 'Ex)
'EN
C (%)
Figure 5: Ablation study for MAE of Hamiltonian H, orbital energy s, and orbital coefficients C for DEQHNet, DEQHNet without the overlap matrix as an input, QHNet, and QHNet with the addition of the overlap matrix as an input, respec- tively.
Conclusion
In essence, the DEQH model fuses DEQs with Hamiltonian learning to obviate the need for iterative DFT computations during the training stage, which are typically required to introduce the self- consistent properties of the Hamiltonian. Our model architecture is meticulously designed to directly capture the fixed-point representation of the Hamiltonian, capitalizing on its intrinsic iterative characteristics accurately reflect the fundamental principles of molecular systems. This innovative strategy propels the DEQH model beyond the traditional confines of predictive machine learning models, transforming it into an iterative solver that refines Hamiltonians. This paradigm shift ushers in a new epoch of computational efficiency and scalability, enabling the model to adeptly process voluminous datasets and master the complexities of elaborate systems. The DEQH model methodology is a universal one, readily adaptable to established machine learning models for predicting Hamiltonians. With QHNeet as the backbone, our DEQHNet has exhibited enhanced predictive accuracy in empirical evaluations on the MD17 and QH9 datasets. We have further delved into the convergence behavior of DEQHNet to elucidate the effectiveness. Our ablation study further reinforces the viability of merging DEQs with neural networks for efficient Hamiltonian learning.
10

---

References
[1] Jörg Behler and Michele Parrinello. Generalized neural-network representation of high- dimensional potential-energy surfaces. Phys. Rev. Lett., 98 98(14):146401, 2007.
[2] Kristof Schütt, Pieter-Jan Kindermans, Huziel Enoc Sauceda Felix, Stefan Chmiela, Alexandre Tkatchenko, and Blaus-Robers Müller. Schnet: A continuous filter convolutional neural network for modeling quantum interactions. Advances in neural information processing systems, 30, 2017.
[3] Kindermans and K-R Müller. Schnet-a deep learning architecture for molecules and materials. J. Chem. Phys., 148(24), 2018.
[4] Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-and translation eqquvvvaaannni neural networks for 3D point clouds. Preprint at http: // aræiv. org/ abs/ 1802. 08219 2018.
[5] Benjamin Coors, Alexandru Paul Condurache, and Andreas Geiger. SphereNet: Learning spher- ical representations for detection and classification in omnidirectional images. In Proceedings of the European conference on computer vision (ECCV), pages 518-533, 2018.
[6] Brandon Anderson, Truong Son Hy, and Risi Kondor. Cormorant: Covariant molecular neural networks. Advances in neural information processing systems, 32, 2019.
[7] Johannes Gasteiger, Janek GroB, and Stephan Günnemann. Directional message passing for molecular graphs. In International Conference on Learning Representations, 2019.
[8] Johannes Gasteiger, Shankari Giri, Johannes T Margraf, and Stephan Günnemann. Fast and uncertainty-aware directional message passing for non-equilibrium molecules. Preprint at http: // arziv. org/ abs/ 2011. 14115 2020.
[9] Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling. SE (3)-transformers: 3D Roto-translation equivariant attention networks. Advances in Neural Information Processing Systems, 33:1970-1981, 2020.
[10] Kristof Schütt, Oliver Unke, and Michael Gastegger. Equivariant message passing for the prediction of tensorial properties and molecular spectra. In International Conference Machine Learning, pages 9377-9387 PMLR, 2021.
[11] Johannes Gasteiger, Florian Becker, and Stephan Günnemann. Gemnet: Universal directional graph neural networks for molecules. Advances in Neural Information Processing Systems, 68629,068002, 2021.
[12] Philipp Thölke and Gianni De Fabritiis. Equivariant transformers for neural network based molecular potentials. In International Conference on Learning Representations, 2021.
[13] Simon Batzner, Albert Musaelian, Lixin Sun, Mario Geiger, Jonathan Mailoa, Mordechai Kornbluth, Nicola Molinari, Tess E Smidt, and Boris Kozinsky. E (3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials. Nat. Commun, 13(1):2453, 2022.
[14] Albert Musaelian, Simon Batzner, Anders Johansson, Lixin Sun, Cameron J Owen, Mordechai Kornbluth, and Boris Kozinsky. Learning local equivariant representations for aarge-scale atomistic dynamics. Nat. Commun. 14(1):579, 2023.
[15] Yi-Lum Liao and Tess Smidt. Equiformer: Equivariant graph attention transformer for 3D atomistic graphs. In The Eleventh International Conference on Learning Representations, 2022.
[16] Ilyes Batatia, David P Kovacs, Gregor Simm, Christoph Ortner, and Gábor Csányi. MACE: Higher order equivariant message passing neural networks for fast and accurate force fields. Advances in Neural Information Processing Systems, 35:11423-1144 1436, 2022.
11

---

[17] Limei Wang, Yi Liu, Yuchao Lin, Haoran Liu, and Shuiwang Ji. ComENet: Towards complete and efficient message passing for 3D molecular graphs. Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[18] Zun Wang, Guoqing Liu, Yichi Zhou, Tong Wang, and Bin Shao. Efficiently incorporating quintuple interactions into geometric deep learning force fields. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[19] Yi-Lun Liao, Brandon M Wood, Abhishek Das, and Tess Smidt. EquiformerV2: Improved equivariant transformer for scaling higher-degree representations. The Twelfth International Conference on Learning Representations.
[20] Yusong Wang, Tong Wang. Shaoning Li, Xinheng He, Mingyu Li, Zun Wang, Nanning Zheng, Bin Shao, and Tie- Yan Liu. Enhancing geometric representations for molecules with equivariant vector-scalar interactive message passing. Nat. Commun., 15(1):313, 2024.
[21] Yunyang Li, Yusong Wang, Lin Huang, Han Yang, Xinran Wei, Jia Zhang, Tong Wang, Zun Wang, Bin Shao, and Tie- Yan Liu. kong-short-range ceessag-passsing A physics-informed framework to capture non-local interaction for scalable molecular dynamics simulation. In The Tweifth International Conference on Learning Representations, 2024.
[22] Kristof T Schütt,, Michael Gastegger, Alexandre Tkatchenko, K-R Müller, and Reinhard Maurer. Unifying machine learning and quantum chemistry with deep neural network for molccular wavefunctions. Nat. Commun., 10(1):5024, 2019.
[23] Oliver Unke, Mihail Bogojeski, Michael Gastegger, Mario Geiger, Tess Smidt, and Klaus- Müller. SE (3)-equivariant prediction of molecular wavefunctions and electronic densities. Advances in Neural Information Processing Systems, 34:14434-14447, 2021.
[24] He Li, Zun Wang, Nianlong Zou, Meng Ye, Runzhang Xu, Xiaoxun Gong, Wenhui Duan, and Yong Xu. Deep-learning density functional theory Hamiltonian for efficient ab initio electronic-structure calculation. Nat. Compul. Sci., 266:3377337 2022.
[25] Xiaoxun Gong, He Li, Nianlong Zou, Runzhang Xu, Wenhui Duan, and Yong Xu. General framework for E (3)-equivariant neural network representation of density functional theory Hamiltonian. Nat. Commun., 14(1):2848, 2023.
[26] Haiyang Yu, Zhao Xu, Xiaofeng Qian, Xiaoning Qian, and Shuiwang Ji. Efficient and equiv- ariant graph networks for predicting quantum Hamiltonian. In International Conference on Machine Learning, pages 40412-4044 PMLR, 2023.
[27] He Zhang, Chang Liu, Zun Wang, Xinran Wei, Siyuan Liu, Nanning Zheng, Bin Shao, and Tie- Yan Liu, Self-consistency training for Sanniity-nnnniina--tttiaa hamiltonian prediction. In Forty-fire first International Conference Machine Learning.
[28] Yang Li, Zechen Tang, Zezhou Chen, Minghui Sun, Boheng Zhao, He Li, Honggeng Tao, Zilong Yuan, Wenhui Duan, and Yong Xu. Neural-network density functional theory based on variational energy minimization. Phys. Rev. Lett., 133(7):076401, 2024.
[29] Gengyuan Hu, Gengchen Wei, Zekun Lou, Philip HS Torr, Wanli Ouyang, Han-sen Zhong, and Chen Lin. Self-consistent validation for machine learning electronic structure. Preprint at 2024.
http: // arziv. org/ abs/ 2402. 10186
[30] Alexander Mathiasen, Hatem Helal, Paul Balanca, Adam Krzywaniak, Ali Parviz, Frederik Hvilshøj, Blazej Banaszewski, Carlo Luschi, and Andrew William Fitzgibbon. Reducing the cost of quantum chemical data by backpropagating through density functional theory, Preprint at ht // arziv. org/ abs/ 2402. 04030| 2024.
[31] Shaojie Bai, Zico Kolter, and Vladlen Koltun. Deep equilibrium models. Advances Neural Information Processing Systems, 32, 2019.
12

---

[32] Haiyang Yu, Meng Liu, Youzhi Luo. Alex Strasser, Xiaofeng Qian, Xiaoning Qian, and Shuiwang Ji. QH9: quantum hamiltonian prediction benchmark for QM9 molecules. Advances in Neural Information Processing Systems, 36, 2024.
[33] Ganesh Hegde and R Chris Bowen. Machine- learned approximations to density functional theory hamiltonians. Sci. Rep., 7(1):42669, 2017.
[34] Yuxiang Wang, He Li, Zechen Tang, Honggeng Tao, Yanzhen Wang, Zilong Yuan, Zezhou Chen, Wenhui Duan, and Yong Xu. Deeph-2: Enhancing deep-learning electronic structure via an equivariant local-coordinate transformer. Preprint at http: // arziv. org/ abs/ 2401. 17015 2024.
[35] Shaojie Bai, Vladlen Koltun, and Zico Kolter. Multiscale deep equilibrium models. Advances in neural information processing systems, 33:5238-5250, 2020.
[36] Zhuoran Qiao, Anders S Christensen, Matthew Welborn, Frederick Manby, Anima Anandku- mar, and Thomas F Miller III. Informing geometric deep learning with electronic interactions accelerate quantum chemistry. Proc. Natl. Acad. Sci., 119631):c22052211 119, 2022.
[37] John Perdew, Kieron Burke, and Matthias Ernzrrhof. Generalized gradient approximation made simple. Phys. Rev. Lett., 77(18)33865, 1996.
[38] Florian Weigend and Reinhart Ahlrichs. Balanced basis sets of split valence, triple zeta valence and quadruple zeta valence quality for H to Rn: Design and assessment of accuracy. Phys. Chem. Chem. Phys., 7(18):327733355 2005.
[39] Frank Neese, Frank Wennmohs, Ute Becker, and Christoph Riplinger. The ORCA quantum chemistry program package. J. Chem. Phys., 152(22), 2020.
[40] T Ozaki and H Kino. Numerical atomic basis orbitals from h to kr. Phys. Rev. B, 69(19)115513, 2004.
[41] Qiming Sun, Xing Zhang, Samragni Banerjee, Peng Bao, Mare Barbry, Nick S Blunt, Nikolay A Bogdanov, George H Booth, Jia Chen, Zhi-Hao Cui, et al. Recent developments the PySCF program package. J. Chem. Phys., 153(2), 2020.
[42] Lars Ruddigkeit, Ruud Van Deursen, Lorenz C Blum, and Lean-Louis Reymond. Enumeration of 166 billion organic small molecules in the chemical universe database GDB-17. J. Chem. Inf. Model., 52(1 (1111)):86428888 2875, 2012.
[43] Raghunathan Ramakrishnan, Pavlo O Dral, Matthias Rupp, and O Anatole Von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Sei. Data, 1(1):1-7, 2014.
[44] Xiangqian Hu and Weitao Yang. Accelerating self-consistent field convergence with the augmented Roothaan-Hall energy function. J. Chem. Phys., 132(5), 2010.
[45] Richard Martin. Electronic structure: basic theory and practical methods. Cambridge university press, 2020.
[46] Jun John Sakurai and Jim Napolitano. Modern quantum mechanics. Cambridge University Press, 2020.
[47] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
[48] Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with Py PoTchh Geometric. Preprint at http: // arziv. org/ abs/ 1903. 02428 2019.
[49] Mario Geiger and Tess Smidt. e3nn: Euclidean neural networks. Preprint at http: // arviv. org/ abs/ 2207. 09453 2022.
[50] Zhengyang Geng and Zico Kolter. Torchdeq: library for deep equilibrium models. Preprint at http: // arziv. org/ abs/ 2310. 18605 2023.
13

---

[51] Daniel GA Smith, Lori A Burns, Andrew C Simmonett, Robert M Parrish, Matthew C Schicber, Raimondas Galvelis, Peter Kraus, Holger Kruse, Roberto Di Remigio, Asem Alenaizan, et al. PS14 1.4: Open-source software for high-throughput quantum chemistry. J. Chem, Phys., 152(18), 2020.
[52] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference Learning Representations, 2018.
14

---

A Supplementary Material
A.1 Electronic structures
The Hamiltonian for a system composed of electrons and nuclei is given by:
H
1/2
Zre²
2
1
Σ
2M,
12
Ze R, R.l"
(15)
2m
R 2
r
2M
i#j
1
where electrons are represented by lowercase subscripts, while nuclei, characterized by their charge Z1 and mass M,, are denoted by uppercase subscripts.
In the general Hamiltonian, the inverse mass of the nuclei, denoted as 1/Mr, stands out as the only term that can be considered "small." This allows to establish perturbation series based on this parameter, which holds broad applicability across the fully interacting electron-nuclei system. Initially, by setting the nuclear mass to infinity, we can disregard the kinetic energy of the nuclei. This leads us to the Born- Norn-Oppenheinme or adiabatic approximation. This approximation proves highly effective for various applications, such as calculating nuclear vibration modes in most solids.
With the omission of nuclear kinetic energy, the fundamental Hamiltonian for the theory of electronic structure simplifies to:
A = + Vox + Vine + E11-
(16)
Adopting Hartree atomic units where h = me 4TED = 1, we can express the kinetic energy operator for the electrons, T as:
T
V
(17)
The external potential acting on the electrons due to the nuclei, Vext, is given by:
Ven
Σ
Vi(In: Rr()-
(18)
1.I
Vi encompasses the electron-electron interactions,
(19)
int
r ry-
which quantifies the electron-electron Coulomb repulsion, a fundamental aspect of electronic in- teractions. The last term, E11, represents the classical interactions between nuclei themselves and encompasses any additional contributions to the system's 's total energy that are not directly related to the description of electron behavior. In this model, the influence of nuclei on the electrons is integrated into fixed "external" potential. This formulation remains applicable even when the straightforward nuclear Coulomb interaction substituted with pseudopotential, which accounts for core electron effects, albeit these potentials are "non-local".
There are two basic independent-particee approaches that may be classified "non-interacting" and "Hartree-Fock" [45) They are similar in that each assumes the electrons are uncorrelated except that they must obey the exclusion principle. However, they are different in that Hartree-Fock includes the dicctron-electron Coulomb interaction in the energy, while neglecting the correlation that is introduced in the true wavefunction due to those interactions. In general, "non-interacting" theories have some effective potential that incorporates some effect of the real interaction, but there is no interaction term explicitly included in the effective Hamiltonian. This approach is often referred to as "Hartree" or "Hartree-like like, after D. R. Hartree who included an average Coulomb interaction in rather heuristic way. More the point of modern calculations, all calculations following the Kohn-Sham method involve non- con-interacting Hamiltonian with an effective potential chosen to incorporate exchange and correlation effects approximately.
According to our comprehensive definition, calculations for non-interacting electrons necessitate solving Schrödinger-lie equation, represented
h22
Here's (r) = 2m
22 + (r)
w/ (x)
E (r),
(20)
15

---

where (r) denotes an effective potential influencing each electron of spin o at position r. To determine the ground state for a collection of non-interacting electrons, one must populate the lowest eigenstates, adhering to the exclusion principle. If the Hamiltonian lacks spin-dependence, the spin states, both up and down, are considered degenerate, effectively doubling the count for these states. The occupation of higher energy eigenstates characterizes the excited states. The primary rationale for employing such mndependent-prttinn equations for electrons in materials is founded on density functional theory.
At finite temperatures, straightforward utilize the general formulas of statistical mechanics to demonstrate that the equilibrium distribution of electrons conforms to the Fermi-Ding (or Bose- Einstein) distribution for occupation numbers as a function of energy. The expectation value, aggregates over many-body states V, each defined by set of occupation numbers n. for individual particle states with energies e.. With each ng being either or and the sum Σ; ng = No, it can be shown that:
(Ô) - f WIIOWN
(21)
where (w)g (l)))) represents the expectation value of the operator Ô for the one-partinn me-particle state v/r and 1 indicates the probability of finding an electron in state i, spin o. Specifically, the Fermi-Diree distribution is given by:
(22)
Ble 1)) +1'
where y represents the Fermi energy the chemical potential of the electrons. The system's energy, for instance, is the weighted sum of the energies of non-interacting particles:
E(T) = (H) = Σ -. f.C.
(23)
1,0
Similar to the general many-body scenario, bongle-boyy density matrix operator can be defined:
Σ f1
(24)
1,
where the expectation value (Ō) is equal Tr(pO). For specific representation involving spin and position, I is expressed as:
p(r,o, = óo.or Σ (orh) (ab?()))
(25)
where the density is the diagonal component:
n (r) = p(r, Σ SIW
(26)
The Hartree-Fock method is another fundamental approach in many-particle theory, which involves constructing properly antisymmetrized determinant wavefunction for predefined number, N, of electrons. The goal is find single determinant that minimizes the total energy of the fully interacting Hamiltonian. In cases where there is no spin-orbit interaction, this wavefunction, denoted as can be expressed as Slater determinant:
Φ (r1, 11) og (2(11,01)
Φ1 (12, 02)
& (ra, 03)
(2 02)
(2 03)
① VN!
23 (3(1),01)
(2
0s 03)
(27)
where each (1(r,, .T) is a particle "spin-orbital comprising spatial function, (xy)), and spin function, az(a),. ). It's important to note that v)x(r)) is spin-independent closed-shell systems, aligning with the "spiierestricted Hartree-Food approximation." These spin-orbitals must not only be linearly independent but also orthonormal to simplify the equations significantly. can be straightforwardly demonstrated that & is normalized to 1.
16

---

Moreover, if the Hamiltonian does not depend on spin or is diagonal in the spin basis or = 11: 4) the expected value of the Hamiltonian using Hartree atomic units with the wavefunction calculated as follows:
f dr
(-2 + Vau(r) / (oar"v (r') r- v/ (r
(0111(0)
Σ
1/ (r) + Eis
6,
Σ
(28)
2
Σ
drdr' vs (r))(r))g (r) 1 " (5)
2
k.j.a
The first term in the Hartree-Foct method consolidates the single- bogge- expectation values by sum- ming the orbitals, while the third and fourth terms address the direct and exchange interactions between electrons, involving double sums. Conventionally, we include the "self-interaction" (where i i). a concept deemed spurious but necessary it cancels out in the aggregation of direct and exchange interactions. When this self-interaction is accounted for, the cumulative effect across all orbitals represents the electron density, and the direct term effectively simplifies the Hartree energy. The "exchange" term specifically interacts between electrons of the same spin, owing to the orthogonality of spin components in orbitals for opposite spins. These interactions, crucial to the energy computation. The essence of the Hartree-Food FFok approach is to minimize the total energy subject to all constraints imposed by the wavefunction's degrees of freedom. This minimization process leverages orthonormality to streamline the equations and employs Lagrange multipliers to ensure this condition is met throughout.
A.2 Real spherical harmonics functions
real basis of spherical harmonics Yi S2 + IR can be defined in terms of their complex analogues Y 52 C by setting
mm - (-1)myy!'m
if m < 0
V2
Yim-
y
m = 0
(29)
|m
+ (-1)mm
m > 0
The Condon-Shortley phase convention is frequently employed to maintain uniformity and con- sistency across the analysis. The corresponding inverse equations defining the complex spherical harmonics Y, 22 + C in terms of the real spherical harmonics Yim S2 - IR are
(Y ml ii
if m 0
//
Y10
if m 0
(30)
(-1) (Yimml + +YY.--mm)
if m > 0
well-established from the analytical solutions of the hydrogen atom that the eigenfunctions corresponding to the angular component of the wave function manifest as spherical harmonics. Intriguingly, when magnetic terms are absent, the solutions to the non-relativistic Schrödinger equation can be rendered real functions. This characteristic underpins the prevalent of real- form basis functions in electronic structure computations, as obviates the need for complex algebra in the software implementations. It is essential to recognize that these real- valued functions occupy an identical functional space to that of their complex counterparts, ensuring no loss of generality or completeness in the solutions.
A.3 Irreps representation and tensor product
Irreps representation The model's equivariant framework harnesses the special orthogonal group SO(3) to represent the 3D rotational symmetries that are fundamental to molecular structures. It utilizes the irreducible representations (irreps) of SO(3), denoted by an integer which resonate with spherical harmonic functions Yim- These harmonics confer rotational attributes upon the feature vectors, granting the model rotational equivariance and allowing for uniform geometric property assessment.
17

---

Tensor product In pursuit of heightened expressiveness within the model. it orchestrates interac- tions among irrep features corresponding to different angular momenta / via the tensor product. This operation combines two irreps with angular momenta l and la to form a new irrep characterized by angular momentum 13, employing Seehsch-Gordan coefficients facilitate an expansion that weighted by "m..mm²
hig..ng
@ hig.m2
Σ
Wmy 11 bi, mny Mag
Cla-ms
(31)
This mathematical formulation enables the synthesis of complex features from simpler ones, capturing the intricate interplay of angular momentum the molecular description.
A.4 Wigner-Eckart theorem
The Wigner- theorem [46], a comerstone of representation theory, articulates that the matrix elements of spherical tensor operators, when projected onto a basis of angular momentum eigenstates, can be factorized into two distinct components: a term invariant to angular momentum orientation and Clebsch-Gordan coefficient. It provides bridge connecting the symmetry transformation groups governing spatial configurations, as applied within the Schrödinger equation framework, to the fundamental conservation principles of energy, momentum, and angular momentum.
Mathematically articulated, the Wigner-Eckart theorem posits that for a given tensor operator T(k and two states with angular momenta j and j'. there exists a reduced matrix element (j) vertj') that is invariant with respect the magnetic quantum numbers m, m', and g. The theorem asserts that for all values of these quantum numbers, the matrix elements of the tensor operator satisfy the relation:
(5''' = (j'mm (kggimmmm || ((k) !1)))
(32)
where T:(40 is the g-th component of the spherical tensor operator (k) of rank k, and (jm) represents an eigenstate of the total angular momentum operator J² and its z-component J2 (j'm' kgljm) is the Clebsch-Gordan coefficient, which mediates the coupling of angular momentum j' with k to yield
j. (i || (()) J") symbolizes the reduced matrix element, a scalar that encapsulates the essence of the tensor operator independent the magnetic quantum numbers. This theorem elegantly decouples the geometric dependencies from the dynamic properties of the matrix elements, thereby simplifying the computation transition amplitudes.
A.5 Proof of the proportionality between on-site three-index overlap integrals and Clebsch-Gordan coefficients
The order-l equivariant embeddings h' using on-site three-index overlap integrals Q:
Q 122,
alm
[ J
Φ mo (r)
2x
Gni exp (-mnn r2) pes pliymia (B. (0,
Gog.t2 exp ( (mmadd2 22 play (0, sp)
Cn,1 exp (-),, r p'y (0, s0 2²2 sin ddrd
Caid exp (-7ny 22) r Cag exp 12) 2 Cni exp (-mmir2) 2' (33)
0
2n
Y Y. y (0.)))))))))) (y", (8, up) sin Addds
dS2Y (f)
x
(2l, + 12 + 1)
4
12
!
l
l
0
4T(21 +
0
0
m1
m2
m
18

---

1 m= m2 m
4
12
where
denotes the Wigner 3j Bj-symbols, which are proportional to the Clebsch-
Gordan coefficients.
A.6
Auxiliary Gaussian-type basis functions
Adhering to the methodology outlined in Ref. [36,, we define the auxiliary Gaussian-type basis functions, and for the sake of brevity, we consider their form the origin, X,4 0:
mm (r) == Ca.t exp(-Tmat
Ynd x²) Yim
(34)
where Cr, is normalization constant such that (.l1"""""(rr))) tions. For numerical experiments considered in this work the scale parameters y are chosen (in atomic units):
dr following standard conven-
7n.==0 128 (0.5)=-1
where n (1,2, 16)
(35)
7 32 (0.25) n- where n 21, ,83
(36)
7n.=2 4.0 (0.25) R-1 where n E 21,2,3, 3,
(37)
Note that the auxiliary basis A is independent of the atomic numbers thus the resulting 1 are of equal length for all chemical elements.
A.7 Deep equilibrium model
Here, we adopt generalized notation for the DEQ 31 function, denoted as f(2, x). This notation encapsulates the earlier specified function f(x, y) activation function, and W, U, and b denote the weight matrix, input weight matrix, and bias vector, respectively. Our objective here determine a stable fixed point = that satisfies the equilibrium condition zx [==",
( Ux + b). where a represents the
Any deep network- wwwwork-irrespettiii irrespective of its depth the intricacy of its connections-can be succinetly encapsulated within the framework of single-layer DEQ model. Remarkably, this encapsulation does not succumb to the typical exponential surge in parameters that is often associated with universal function approximation theorems for single-layer models. In essence, single-layer DEQ model can represent the functional capacity of any network without necessitating an increase in the number of parameters. Consider a conventional deep network described by the composition of two functions, denoted as y 92 (g (z)). This construct can be seamlessly transmuted into single-layer DEQ model by amalgamating all intermediate variables derived from the computation into an extended vector,
[(2,)) / (
21 Zig
[x)
(38)
(2((1))
At an equilibrium point 2" of this function,
f( x) = = =1 =2 (2())
(39)
We can concatenate all intermediate results of computational graph into the vector z, and designate the function f as the operator that applies the "subsequent" computation in the graph to each of these elements. This theoretical framework unequivocally demonstrates the representational strength of single DEQ layer.
To elucidate the principles of implicit backpropagation tailored to DEQ models-and by extension, to any fixed-point iterative layer-we will primarily concentrate on the DEQ model's specific form herein. Our objective is to compute the vector-Jacobian product 0==()) yTy for given vector where (-) symbolizes any variable with respect which we seek to differentiate the fixed point, all of which influence the determined fixed point z". Differentiating both sides of the fixed-point equation yields:
0z*(.) af(=*(),z) 0(-)
0f(=", x)xx))00 02)())
0f(=",x)
+
(40)
0(.)
Oz
0(-)
0(-)
19

---

where 2" (-) denotes the case where 2" is treated as an implicit function of the differentiated quantity. and x* alone signifies the equilibrium value. The second equality arises by applying the multivariate chain rule. Rearranging terms, we arrive at an explicit expression for the Jacobian:
dz)()) = (I = a(-)
Of(=". x) )-1 af(="x))
(41)
Oz-
a(.)
where the terms on the right-hand side can be computed using conventional automatic differentiation. To compute the rector-Jacobian product,
(0000)
(1- 0f(=",0)
T
TT
a(.)
a()
2z+
y.
(42)
In practice, the focal term the solution to the linear system (abbreviated as g):
g 0f(2",0) Oz
-T
y,
(43)
which can be expressed as fixed-point equation:
(0f(et))) 0z*
T
g
+
(44)
The convergence of this equation via forward iteration hinges on the stability of the Jacobian 0fi="..) which also underlies the local stability of the forward iterative process.
0%"
The derivation of the vector-Jacobian product for a DEQ layer can be distilled into a two-step procedure:
Solve the Fixed-Point Equation g = (0f(2" .x)
g + y. This can be done either through direct analytical inversion or, more commonly, by employing an iterative method that neces- [af(="
Oz*
7"
Such multiplications can be expediently performed using standard automatic differentiation, as they are vector-Jacobian products in their own right.
sitates only the multiplications by
02"
Calculate the final vector-Jacobian product (0z-
g. This step also leverages the capabilities of conventional automatic differentiation, as involves the compu- tation of another vector-Jacobian product.
T
y af(=",.) T
3(-)
3()
By executing these steps, one can integrate DEQ layers into the backpropagation algorithm, thereby harnessing the power of implicit differentiation to effectively train deep equilibrium models.
A.8 Metrics
To evaluate the accuracy and computational efficiency of the predicted Hamiltonian matrix, we adopt a set of metrics as demonstrated in Ref. [32 to evaluate the model's 's performance:
MAE Hamiltonian matrix и The MAE against DFT-computed reference data, considering both diagonal and off-diagonal blocks that represent intra- and inter-atomic interactions, respectively. Given the sparsity induced by distant atom pairs larger molecules, we separately assess the MAE for diagonal and off-diagonal blocks along with the total MAE of the matrix.
MAE of occupied orbital energies The MAE for occupied orbital energies, including HOMO and LUMO levels derived from the predicted and reference Hamiltonian matrices, is used to measure the matrix's predictive accuracy for these critical properties.
Cosine similarity of orbital coefficients & To determine the similarity between predicted and reference electronic wavefunctions, we compute the cosine similarity of the coefficients for occupied molecular orbitals, which are crucial for inferring chemical properties.
20

---

Table 3: The performance of DFT calculation accelcration.
Training Dataset
DFT initialization
Metric
QHNet
DEQHNet
Optimal ratio
0.057 0.004
0.060 0.004
le
Achieved ratio
0.395 = 0.030
0.363 = 0.074
Error-level ratio
0.635 0.039
0.683 0.039
QH9-stable-id
Optimal ratio
0.102 0.005
0.113 0.006
minao
Achieved ratio
0.706 = 0.031
0.681 = 0.129
Error-level ratio
0.408 = 0.025
0.459 0.031
Optimal ratio
0.057 0.004
0.060 0.004
le
Achieved ratio
0.400 0.030
0.359 0.031
Error-level ratio
0.620 0.037
0.678 0.038
sta-stable-ood
Optimal ratio
0.102 + 0.005
0.113 0.006
minao
Achieved ratio
0.715 - 0.033
0.674 0.045
Frror-level ratio
0,406 0.021
0.454 + 0.029
Optimal ratio
0.056 = 0.006
0.056 = 0.006
le
Achieved ratio
0.392 + 0.036
0.373 = 0.037
Error-level ratio
0.648 0.041
0.673 0.037
0H9-dymamie-100k-yee
Optimal ratio
0.098 + 0.008
0.098 = 0,008
minao
Achieved ratio
0.679 + 0.041
0.646 = 0.043
Error-level ratio
0.443 0.044
0.475 0.050
Optimal ratio
0.056 + 0.006
0.056 = 0.006
le
Achieved ratio +
0.512 0.138
0.426 = 0.081
Error-level ratio
0.622 + 0.048
0.644 = 0.048
QH9-dynamic-1000 100kmoo
Optimal ratio
0.098 + 0.008
0.098 0.008
minao
Achieved ratio 4
0.882 0.217
0.736 = 0.128
Error-level ratio
0,406 + 0.066
0.436 = 0.050
Acceleration ratio Acceleration ratios, such as the achieved ratio and the error-level ratio, could be used to evaluate how well the predicted Hamiltonian matrix speeds up DFT calculations. These ratios compare the number of optimization steps needed when using the predicted matrix initial guesses, and the steps required to reach a comparable error level in the DFT SCF cycle.
traditional
A.9 Accelerate ratio of DEQHNet
The DEQHNet, after being trained the QH9 dataset, undergoes evaluation meticulously selected subset of molecules. This subset consists of 50 molecules that have been randomly chosen from the overlapping elements of the QH9-stable and QH9-dynamie's respective test subsets. The intersection ensures that the evaluation is conducted on a common ground, facilitating a direct comparison of the performance results between the stable and dynamic regimes. As illustrated in T the study underscores the feasibility of leveraging DEQHNet as initial seeds for electronic structure computations, which can potentially reduce the time to convergence significantly.
A.10 Explanation of the error in orbital energy
In the experiments with MD17 and QH9, we observed a decrease in the Mean Absolute Error (MAE) of the Hamiltonian, but the MAE of the orbital energy did not show corresponding positive trend. To investigate this, we conducted additional experiments. We randomly selected a molecule, added Hermitian Gaussian noise matrix its Hamiltonian, and solved the corresponding generalized cigenvalue equation. As shown in Fig. 6 it can be seen that as the noise on the Hamiltonian gradually increases, the range of error in the orbital energy becomes larger. The errors in the orbital energy of QHNet and DEQHNet on MD17 and QH9 are all within the range shown in the figure. Only when the MAE of the Hamiltonian is sufficiently small can it be ensured that the corresponding orbital energy eтror is also small. In the comparison between QHNet and PhiSNet, we observed similar situations. This somewhat suggests that using existing Hamiltonian error measurements cannot definitively reflect the errors in the eigenvalues. These evidence suggests that is nontrivial and long-standing challenge in the domain Hamiltonian prediction to design proper metric on the matrix space so that reflects the metric of derived quantities e.g. energy, for which we will investigate in future work.
21

---

3500
3000
2500
(10-$EA)
2000
1500
1000
500
50
75
100
125
150
175
200
ΔH (10-*En)
Figure 6: The variation in the error of orbital energy with the injection of Hermitian Gaussian noise to the Hamiltonian.
Table 4: Hyperparameters of DEQHNet for different datasets.
Water
Dhanel
Makendichssk
Usail
Subbeiz
StaHoooa
Dynamic-ge 99.00099992000000
Dynanioms
25.00130011473
71.0089000100100
Osla Tammsis larmssss
liges
Seadh
22
32
Loninng rat (ER)
1o3
Ie-3
1o.3
Soid
Se-4
So-d
ESGA
Ira
Iine
Pake
EMA strt epech
NA
NA
NA
(Blcooadd)
A.11 Settings of experiments
The overarching framework is implemented using PyTorch [47], PyTorch Geometric [48] e3nn 49
and Torchdeq (50) libraries. The computation of on-site three-index overlap integrals is conducted by
Psi4 software [51] and subsequently converted to adhere to the conventions of PySCF [41]. The loss function for training DEQHNet model is the summation of the Frobenius absolute eтror of the matrix of Hamiltonians,
and L.1 norm of the
L
N Σ H.s H2 +
Σ 1
His
H
(45)
N2
where N the number of elements in Hamiltonian matrix. H and Ii denote the predicted and ground-truth Hamiltonian respectively. Adam optimizer [52) was adopted. DEQHNet model was trained on single 16G Nvidia Tesla V100 GPU for MD17 dataset and single 80G Nvidia Tesla A100 GPU for QH9 dataset. It is noteworthy that the iterative process inherent in DEQ may impact training efficiency to certain extent. our experiments with the MD17 dataset, we observed that the model eventually converges after only 2-3 DEQ iterations. Therefore, to enhance training efficiency, we set the maximum number of iterations for both the forward and backward passes of DEQ to 3 in our experiments with the QH9 dataset. With this configuration, the training time on the QH9 data was approximately 1.5 times that QHNet. Furthermore, detailed settings of hyperparameters are summarized in the Table
A.12 Rethinking and Prospects
The Hamiltonian exhibits self-consistent iterative properties. Specifically, the DFT solution process, the information from the previously obtained Hamiltonian is used to construct the current Hamiltonian
22

---

for the current DFT solution. This process is repeated until certain criteria meet set threshold. We hypothesize that introducing DEQ enables the network learn the iterative mechanism of solving the Hamiltonian, rather than directly learning the solution to the Hamiltonian. This shift could allow the network to better capture inherent physical laws, thereby improving generalizability. Compared to models without self self-consisteney, DEQ introduces additional orbital information to the network input. This, coupled with the introduction of scientific priors, can expedite DFT convergence even when the model's output is simply used as the initial value for DFT. In contrast with methods that introduce self-consistency (which often include DFT computations in the loss function during model training), method offers an architectural enhancement. The DEQH model does not directly include DFT computations, leading to a more efficient computational process. Empirically the dataset and molecular size are relatively small and the cost of DFT computations during training is acceptable, previous methods introducing self-consistency might outperform ours (as suggested by DEQHNet's results water, where DEQHNet requires more data). However, when the dataset is large enough, or when it includes large molecules where DFT computation cost is prohibitive, the benefits of the DEQH model become more significant.
The DEQH model a versatile approach that can be seamlessly integrated with current machine learning models for Hamiltonian prediction. In this paper, opted for QHNet (26 over PhiSNet (2 due to the current implementation limitation stems from PhiSNet's matrix prediction module, which is tailored to predict matrices of fixed size for identical molecules [3].. Additionally, based on the ablation studies, we also posit that providing the overlap matrix could enable training data whose DFT labels are associated with varying basis sets, whereas models lacking it may be restricted to consistent DFT-level datasets. Furthermore, the DEQH model employs a method that is entirely orthogonal to the inclusion of iterative DFT processes during training to achieve the self-consistency of the Hamiltonian an architectural innovation within the model. Incorporating DFT iterations into the existing Hamiltonian training introduces more explicit physical prior [ 28 29 30 requires larger dataset enable the model to learn the iterative solving process. However, for cases involving large datasets and larger molecules, the DEQH model may be more appropriate. This not only offers the possibility of direct integration with off-the- the- training methodologies that incorporate self-consistency but may also assist in enhancing aspects such as model generalizability.
PhiSNet being restricted to single-molecule support. This
compared to the DEQH model, which
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have provided cases, statistical numbers for our motivation, and also extensive experiments for verification to support our claim.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have discussed the limitation in Supplementary Material A.12 and provided the potential solutions.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
24

---

Answer:
[Yes]
Justification:
We have provided the proof in Supplementary Material A.5
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have provided extensive experimental results and all the information needed to reproduce experimental results in Supplementary Material A.I1 We open source our implementation at https:/// http:/ggtththhb/e. com/ Zun Wang/DEQHllet,
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
25

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We open source implementation at htttts::/ com Zun Wang/ DEQ
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis.cc
cublicc/qudees/Codeuubbii www.eeeeuuumissioooooooooooo for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ wwtps/cccpbplgguuuuaeeeeae
Stleee//CoSSSSbmsssS.Si. www.ssssiicgcyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have provided all the information needed reproduce experimental results in Supplementary Material A.11
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The mean and variance of the experimental results are reported in Fig. and Table. 3
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
26

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have provided all the information in Supplementary Material A.1I
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification:
We have reviewed the NeurIPS Code of Ethies.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
27

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
All the data can be found publicly, and open source implementation https://github. com Zun Wang/ DEQQHNtt
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
28

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
29

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "ZROsA5iomK", "reviewer": "Reviewer_eJGc", "summary": "This paper introduces the DEQH (Deep Equilibrium Quantum Hamiltonian) model, which integrates Deep Equilibrium Models (DEQs) for predicting quantum Hamiltonians. By incorporating DEQs, the model captures the self-consistency of Hamiltonians without needing iterative Density Functional Theory (DFT) calculations during training, enhancing computational efficiency. DEQHNet, a specific implementation, demonstrates improvements in prediction accuracy on the MD17 and QM9 datasets. The model acts as both a predictor and solver, iteratively refining the Hamiltonian to achieve self-consistency. Ablation studies further validate the effectiveness of this approach.", "questions": "What are the runtime and memory usage for DEQHNet compared to previous methods?\n\nWhat are the theoretical assumptions underlying the DEQH model, and how do they impact the generalizability of the results?", "limitations": "The limitations are discussed.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The DEQH model proposed eliminates the need for iterative DFT calculations during training, which reduces computational overhead.\n\n2. The DEQHNet model demonstrates improved accuracy in predicting Hamiltonians on the MD17 and QM9 datasets.\n\n3. The model inherently captures the self-consistency required for accurate Hamiltonian prediction.\n\n4. This paper includes ablation studies to analyze the contribution of different components of the model.\n\n5. The paper shows quick convergence of the DEQHNet model.", "weaknesses": "1. The presentation of the paper could be improved. In this paper, the integration of Deep Equilibrium Models (DEQs) with the Hamiltonian solver is presented in a quite technical way. Additional visual illustrations and diagrams could help clarify the workflow of the DEQH model and its components. \n\n2. This paper primarily compares DEQHNet with QHNet. Benchmarking with additional methods could help better evaluate the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are immensely grateful for your assessments. We will address each point in our response accordingly.\n\n## Weakness 1:\nThanks for your feedback and suggestions. We provide a PDF document in the global rebuttal section, which includes a figure delineating the distinction between the off-the-shelf model used for Hamiltonian prediction and the DEQH model. We hope this visual aid will enable readers to gain a clearer and more intuitive understanding.\n\nFurthermore, in the revised version, we will include additional diagrams and illustrations to better depict the workflow of the DEQH model and its components. This will include detailed flowcharts and schematic representations that highlight the key processes and interactions within the model. We are grateful for your insights, which have been instrumental in improving the clarity and quality of our work.\n\n## Weakness 2:\nThank you for your valuable feedback. We agree that adding more benchmark results would provide a more comprehensive evaluation of our DEQHNet model's performance. In response to your suggestion, we include additional benchmark results for SchNOrb [1], PhiSNet [2], and DeepH [3] on the MD17 dataset as follows.\n\n|Dataset|Model|H[10$^{-6} E_h$] $\\downarrow$|$\\epsilon$ [$10^{-6}E_h$] $\\downarrow$|$\\psi$ [$10^{-2}$] $\\uparrow$|\n|--|--|--|--|--|\n|Water|SchNOrb|165.4|279.3|100.00|\n||QHNet|10.79|33.76|99.99|\n||PhiSNet (ori)|17.59|85.53|100.00|\n||PhiSNet (reproduce)|15.67|-|99.94|\n||DeepH|38.51|-|-|\n||DEQHNet|36.07|335.86|99.99|\n|Ethanol|SchNOrb|187.4|334.4|100.00|\n||QHNet|20.91|81.03|99.99|\n||PhiSNet (ori)|12.15|62.75|100.00|\n||PhiSNet (reproduce)|20.09|102.04|99.81|\n||DeepH|22.09|-|-|\n||DEQHNet|18.73|106.94|100.00|\n|Malonaldehyde|SchNOrb|191.1|400.6|99.00|\n||QHNet|21.52|82.12|99.92|\n||PhiSNet (ori)|12.32|73.50|100.00|\n||PhiSNet (reproduce)|21.31|100.60|99.89|\n||DeepH|20.10|-|-|\n||DEQHNet|17.97|93.79|99.90|\n|Uracil|SchNOrb|227.8|1760|90.00|\n||QHNet|20.12|113.44|99.89|\n||PhiSNet (ori)|10.73|84.03|100.00|\n||PhiSNet (reproduce)|18.65|143.36|99.86|\n||DeepH|17.27|-|-|\n||DEQHNet|15.07|107.49|99.89|\n\nWe would like to note that due to the design of DeepH, the data used is re-labeled by OpenMX [4], which may lead to different results compared to other models. Furthermore, considering the high training cost of PhiSNet, we also provide PhiSNet results from the QHNet article for a more balanced and comprehensive comparison.\n\nThese results will be incorporated into the revised manuscript to provide a more robust and thorough evaluation of DEQHNet, thereby offering readers a clearer understanding of its performance relative to other state-of-the-art methods.\n\n[1] Schütt K T, Gastegger M, Tkatchenko A, et al. Unifying machine learning and quantum chemistry with a deep neural network for molecular wavefunctions[J]. Nature communications, 2019, 10(1): 5024.\n\n[2] Unke O, Bogojeski M, Gastegger M, et al. SE (3)-equivariant prediction of molecular wavefunctions and electronic densities[J]. Advances in Neural Information Processing Systems, 2021, 34: 14434-14447.\n\n[3] Li H, Wang Z, Zou N, et al. Deep-learning density functional theory Hamiltonian for efficient ab initio electronic-structure calculation[J]. Nature Computational Science, 2022, 2(6): 367-377.\n\n[4] Ozaki T, Kino H. Numerical atomic basis orbitals from H to Kr[J]. Physical Review B, 2004, 69(19): 195113.\n\n## Question 1:\nThanks for your question regarding the runtime and memory usage of DEQHNet compared to previous methods. As discussed in Section A.10 of our Supplementary Material, the training time of DEQHNet on the QH9 dataset is approximately 1.5 times that of QHNet. This is primarily due to the need for DEQ models to perform fixed-point iteration solving, which results in a higher runtime compared to models that do not incorporate a self-consistency mechanism.\n\nRegarding memory usage, DEQHNet requires extra blocks to process node features constructed from the Hamiltonian and overlap matrix inputs. This results in a slightly larger memory footprint compared to previous network structures. In our experiments, DEQHNet only had a 28.5% increase in parameters compared to QHNet, demonstrating that it improved accuracy without a significant increase in parameters.\n\nWe are currently quantifying these requirements more precisely and will include a detailed comparison of memory usage in the revised manuscript.\n\n## Question 2:\nThanks for your insightful question. The theoretical foundations of our DEQH model are rooted in the common practice in Density Functional Theory (DFT) of iteratively solving the electronic Schrödinger equation. The process begins by initializing a Hamiltonian, solving the generalized eigenvalue problem, and then constructing a new Hamiltonian for the next iteration. We treat this iterative property of the Hamiltonian as a fixed-point iteration problem. This allows us to leverage the characteristics of DEQ in solving fixed-point problems to impose self-consistency on the network for Hamiltonian prediction. \n\nThe results from the QH9-stable-ood experiment demonstrate that our approach can significantly reduce the Mean Absolute Error (MAE) of the Hamiltonian, suggesting an enhanced capability for generalization. Additionally, the DFT Acceleration ratio presented in Appendix A.8 further reflects the generalizability of our model to a certain extent. We hypothesize that introducing DEQ enables the network to learn the iterative mechanism of solving the Hamiltonian, rather than directly learning the solution to the Hamiltonian. This shift could allow the network to better capture inherent physical laws, thereby improving generalizability.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zZE2GWnNvi", "reviewer": "Reviewer_Tf4v", "summary": "The authors introduce DEQH model, which combines deep equilibrium models with existing ML models to predict quantum Hamiltonian, and the author adopt QHNet as the backbone and further develop DEQHNet.  The authors evaluate the proposed method on benchmarks MD17 and QM9, the results show some effectiveness.", "questions": "1. Why is the model performance so much worse than the baseline in terms of energy? Has the author explored the reasons?\n\n2. Is the method proposed by the author extensible and can it be applied to other existing ML models?", "limitations": "The authors have discussed the limitations.", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The idea of incorporating the DEQs is interesting, which have some kind of intrinsic principles as the authors demonstrated in the paper.\n\n2. When compared with QHNet, the DEQHNet shows advantages.", "weaknesses": "1. Though I am relatively familiar with ML for Hamiltonian prediction and have gained some knowledge of DEQ, it is still not easy for me to understand the main part of this paper.\n\n2. This work only compare with one baseline named QHNet, there are some other works which can also be baselines [1] [2] [3].\n\n3. Source code is not available.\n\n[1] Unke O, Bogojeski M, Gastegger M, et al. SE (3)-equivariant prediction of molecular wavefunctions and electronic densities[J]. Advances in Neural Information Processing Systems, 2021, 34: 14434-14447.\n\n[2] Gong X, Li H, Zou N, et al. General framework for E (3)-equivariant neural network representation of density functional theory Hamiltonian[J]. Nature Communications, 2023, 14(1): 2848.\n\n[3] Wang Y, Li H, Tang Z, et al. DeepH-2: Enhancing deep-learning electronic structure via an equivariant local-coordinate transformer[J]. arXiv preprint arXiv:2401.17015, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review. We will address each point in our response accordingly.\n\n## Weakness 1:\nThank you for your feedback. The DEQH model is designed to predict the Hamiltonian. Given the self-consistent iterative properties of the Hamiltonian, we regard these properties as a fixed-point iteration problem, which inspires us to use DEQ to introduce self-consistency into existing models for Hamiltonian prediction.\n\nWe provide a PDF document in the global rebuttal section, which includes a figure delineating the distinction between the off-the-shelf model for Hamiltonian prediction and the DEQH model. Furthermore, we will revise our manuscript to simplify the technical language and include more background information about DEQs and Hamiltonian prediction. We will also add further explanations about the integration of these two components and how this combination contributes to the overall model. We hope that these revisions will make our paper more comprehensible to readers with various backgrounds.\n\nWe appreciate your feedback and will make sure to address this concern in the revised version of our paper.\n\n## Weakness 2:\nThanks for your insightful suggestion. Although DEQH model could be extended to periodic systems, DEQHNet, being built on the foundation of QHNet, is tailored for non-periodic systems. Hence, a direct comparison with DeepH-E3 and DeepH-2 may not be suitable.\n\nIn response to your feedback, we include additional benchmark results for SchNOrb [1], PhiSNet [2], and DeepH [3] on the MD17 dataset as follows.\n\n|Dataset|Model|H[10$^{-6} E_h$] $\\downarrow$|$\\epsilon$ [$10^{-6}E_h$] $\\downarrow$|$\\psi$ [$10^{-2}$] $\\uparrow$|\n|--|--|--|--|--|\n|Water|SchNOrb|165.4|279.3|100.00|\n||QHNet|10.79|33.76|99.99|\n||PhiSNet (ori)|17.59|85.53|100.00|\n||PhiSNet (reproduce)|15.67|-|99.94|\n||DeepH|38.51|-|-|\n||DEQHNet|36.07|335.86|99.99|\n|Ethanol|SchNOrb|187.4|334.4|100.00|\n||QHNet|20.91|81.03|99.99|\n||PhiSNet (ori)|12.15|62.75|100.00|\n||PhiSNet (reproduce)|20.09|102.04|99.81|\n||DeepH|22.09|-|-|\n||DEQHNet|18.73|106.94|100.00|\n|Malonaldehyde|SchNOrb|191.1|400.6|99.00|\n||QHNet|21.52|82.12|99.92|\n||PhiSNet (ori)|12.32|73.50|100.00|\n||PhiSNet (reproduce)|21.31|100.60|99.89|\n||DeepH|20.10|-|-|\n||DEQHNet|17.97|93.79|99.90|\n|Uracil|SchNOrb|227.8|1760|90.00|\n||QHNet|20.12|113.44|99.89|\n||PhiSNet (ori)|10.73|84.03|100.00|\n||PhiSNet (reproduce)|18.65|143.36|99.86|\n||DeepH|17.27|-|-|\n||DEQHNet|15.07|107.49|99.89|\n\nHowever, it should be noted that the data used in DeepH is re-labeled by OpenMX[4]. Furthermore, considering the high training cost of the original PhiSNet, we provide PhiSNet results from the QHNet paper for a more balanced comparison.\n\nWe appreciate your suggestion and will make sure to incorporate these benchmark results in the revised version of our paper.\n\n[1] Schütt K T, Gastegger M, Tkatchenko A, et al. Unifying machine learning and quantum chemistry with a deep neural network for molecular wavefunctions[J]. Nature communications, 2019, 10(1): 5024.\n\n[2] Unke O, Bogojeski M, Gastegger M, et al. SE (3)-equivariant prediction of molecular wavefunctions and electronic densities[J]. Advances in Neural Information Processing Systems, 2021, 34: 14434-14447.\n\n[3] Li H, Wang Z, Zou N, et al. Deep-learning density functional theory Hamiltonian for efficient ab initio electronic-structure calculation[J]. Nature Computational Science, 2022, 2(6): 367-377.\n\n[4] Ozaki T, Kino H. Numerical atomic basis orbitals from H to Kr[J]. Physical Review B, 2004, 69(19): 195113.\n\n## Weakness 3:\nThanks for your feedback. We understand the importance of making source code available for reproducibility and transparency in research. We have prepared an anonymous link (https://anonymous.4open.science/r/nips-rebuttal-80C4/) for the review.\n\n## Question 1:\nThanks for your question. In Section A.9 of Supplementary Material, we discussed this difference in detail. We randomly selected a molecule, added a symmetrized (Hamiltonian matrices need to be symmetric) Gaussian noise matrix, and solved the corresponding generalized eigenvalue equation. As shown in Fig. 5, when the noise on the Hamiltonian gradually increased, the error range of orbital energy also increased. The orbital energy errors we reported for both QHNet and DEQHNet on the MD17 and QH9 datasets fall within the range shown in the figure, which suggests that the experimental results are reasonable. This observation implies that only when the Mean Absolute Error (MAE) of the Hamiltonian is sufficiently small can we ensure that the corresponding orbital energy deviation is also small. Additionally, we repeated the experiments on the MD17 dataset and found that when the Hamiltonian MAE of the model is at the same level, it is possible to obtain both a lower and a higher orbital energy MAE than the baseline. In the comparison between QHNet and PhiSNet, we observed similar situations. This somewhat suggests that using existing Hamiltonian error measurements cannot definitively reflect the errors in the eigenvalues. This is a nontrivial, separate research question that we plan to investigate in the future.\n\nWe will elaborate on this point more clearly in the revised version of our paper to provide a comprehensive understanding of the model's performance.\n\n## Question 2:\nThanks for your question. Indeed, our approach is designed to be highly adaptable and can be applied to off-the-shelf machine learning models. As stated in the introduction, our method introduces self-consistency to off-the-shelf models used for Hamiltonian prediction through the use of Deep Equilibrium (DEQ) models. To incorporate this into existing models, one simply needs to add a block that processes node features constructed from the Hamiltonian and overlap matrix inputs. This makes it relatively straightforward to combine any existing model with DEQ.\n\nWe believe this adaptability is one of the key strengths of our method and will further highlight this in our revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "s2Vdq8kb4U", "reviewer": "Reviewer_DtX8", "summary": "The paper introduces a novel neural network architecture DEQH, extending deep equilibrium models (DEQs) to improve predictions of quantum Hamiltonians. The architecture constrains solutions to ensure self-consistency of the Hamiltonian, thereby improve generalization capability and test accuracy.", "questions": "How do we expect DEQH to compare to methods that ensure self-consistency through integration? Is there are trade-off between performance and computational cost, or do we expect the method to be better overall. Why?", "limitations": "The paper proposes a novel way to embed self-consistency into neural network architectures for quantum Hamiltonian prediction and provides empirical evidence of benefits.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Constraining the network to obey self-consistency is natural thing to do and matches the physical constraints of the problem at hand, without the need for costly DFT calculations.\n\nThe paper proposes an elegant parameterization. Further, it provides an extensive empirical analysis showing the practical benefits of the proposed approach on several datasets.", "weaknesses": "As mentioned, DEQH model acts fundamentally as a solver, iteratively determining the Hamiltonian with fixed point iterations. It is not directly clear to me why this would necessarily be better, or how this compares, to methods that integrating frameworks that also ensure self-consistency. Why would we expect to be better both computationally and in terms of performance, or is there a clear trade-off?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's recognition of the innovation and significance of our work and will address each point in our response accordingly.\n\n## Weakness:\nThank you for your insightful question. The Hamiltonian exhibits self-consistent iterative properties. Specifically, in the DFT solution process, the information from the previously obtained Hamiltonian is used to construct the current Hamiltonian for the current DFT solution. This process is repeated until certain criteria meet a set threshold. We hypothesize that introducing DEQ enables the network to learn the iterative mechanism of solving the Hamiltonian, rather than directly learning the solution to the Hamiltonian. This shift could allow the network to better capture inherent physical laws, thereby improving generalizability. Compared to models without self-consistency, DEQ introduces additional orbital information to the network input. This, coupled with the introduction of scientific priors, can expedite DFT convergence even when the model's output is simply used as the initial value for DFT (as shown in Appendix A.8). In contrast with methods that introduce self-consistency (which often include DFT computations in the loss function during model training), our method offers an architectural enhancement. The DEQH model does not directly include DFT computations, leading to a more efficient computational process. Empirically, if the dataset and molecular size are relatively small and the cost of DFT computations during training is acceptable, previous methods introducing self-consistency might outperform ours (as suggested by DEQHNet's results on water, where DEQHNet requires more data). However, when the dataset is large enough, or when it includes large molecules where DFT computation cost is prohibitive, the benefits of the DEQH model become more significant. As mentioned in Appendix A.11, since these methods are orthogonal, combining them could be a promising direction for future exploration.\n\nWe will make sure to discuss these points more clearly in the revised manuscript.\n\n## Questions:\nThank you for your insightful question. Compared to works introducing self-consistency, our method enhances the model architecture. As mentioned in the related works section of our paper, several methods introduce this attribute during the training process at the loss level, requiring direct DFT computation. In contrast, the DEQH model does not directly involve DFT computation, leading to more efficient computational performance. Empirically, if the data volume and molecular size are relatively small, and the DFT computational cost during training is acceptable, previous works introducing self-consistency might perform better (as suggested by DEQHNet results on water, where DEQHNet requires a larger data volume). However, when the data volume is sufficient, or the data contains larger molecules making the DFT computation cost too high, the improvements offered by the DEQH model become more substantial. As we mentioned in Appendix A.11, since these methods are orthogonal, combining them is a direction worth exploring.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yn81IImMTz", "reviewer": "Reviewer_mPEr", "summary": "Paper presents DEQH model which is a Deep Equilibrium Model to solve quantum Hamiltonians. DEQH uses the deep equilibrium model because it converges to a fixed point, which matches the self-consistent nature of Hamiltonian solving. Paper presents architecture based on the QHNet, results and comparison with QHNet for the MD17 and QH9 datasets, and convergence and ablation study.", "questions": "see above.", "limitations": "A limitation I see is that DFT ground truth calculation is still used in the loss. Since DEQH model is a Hamiltonian solver, it would be nice to see how it performs using a self-consistency loss. It seems that avoiding having to run many DFT calculations would be a big improvement, whereas the current results of paper show DEQH gets lower error results than previous model but still needs DFT data and requires 1.5x training time.", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "Paper addresses a significant problem of computational chemistry. Paper is original and novel in integrating deep equilibrium model with Hamiltonian solving (combining fixed point convergence and self-consistency property makes sense). Paper has good set of experiments comparing with an alternative state of the art model and reaches better results than other model.", "weaknesses": "It seems there are differences between DEQH and QHNet, including the input of the predicted H, but the differences are not clearly highlighted in the paper. this makes the comparison harder to interpret.\n\nThe result of H, Psi errors being low while epsilon errors are high for DEQH is strange. What is the meaning of H, epsilon, and Psi, and why is the epsilon error higher in many cases? No error bars on tables and ablation study, makes results harder to interpret.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review. We will address each point in our response accordingly.\n\n## Weakness 1:\nThank you for your feedback. DEQH is a general method specifically devised to instill self-consistency into pre-existing models, while QHNet is a distinct network that predicts the Hamiltonian directly. Specifically, DEQH functions as a Hamiltonian solver, learning the iterative solving process $H^* = f(H^*, Z, R)$, while QHNet operates as a predictor $H = f(Z, R)$, where $H, Z, R$ denote Hamiltonian matrix, atomic number, and atomic coordinates resepctively. f represents a neural network and $H^*$ denotes the fixed point of the equation. In our research, we utilized QHNet as the fundamental backbone and incorporated DEQH into it, thereby creating DEQHNet. As mentioned in the Supplementary Material A.11, we opted for QHNet over PhiSNet due to the current implementation of PhiSNet being restricted to single-molecule support. This limitation stems from PhiSNet’s matrix prediction module, which is tailored to predict matrices of fixed size for identical molecules.\n\nDEQH introduces self-consistency by approaching the problem of Hamiltonian solution from the vantage point of a fixed-point iteration, which necessitates the inclusion of the Hamiltonian in the network's input. To ensure the broad applicability of the Hamiltonian across various graph neural networks, it must be transformed into signals on the graph. This is demonstrated in Section 4.1, where we transposed the Hamiltonian, overlap matrix, and other equivalent tensors into invariant node features.\n\nFurthermore, we provide a PDF document in the global rebuttal section, which includes a figure delineating the distinction between the off-the-shelf model used for Hamiltonian prediction and the DEQH model. We understand the importance of clearly communicating these differences and will further elaborate on these points in the revised version of our paper.\n\n## Weakness 2:\nThank you for your question. In Appendix A.7, we provide detailed definitions of the metrics. H refers to the Mean Absolute Error (MAE) of the Hamiltonian matrix, while epsilon and Psi are the eigenvalues (orbital energy) and eigenvectors (orbital coefficients) obtained by solving generalized eigenvalue problems. The metrics for epsilon and Psi are the MAE of the orbital energies and the cosine similarity of the orbital coefficients, respectively.\n\nWe explain in Appendix A.9 why epsilon is higher than the baseline in many cases. We selected a molecule randomly, added a Hermitian-Gaussian noise matrix to its Hamiltonian, and solved the corresponding generalized eigenvalue equation. We observed that as the noise on the Hamiltonian increased, the range of errors in the orbital energy also increased. The orbital energy errors reported for both QHNet and DEQHNet on the MD17 and QH9 datasets fall within this range, suggesting the experimental results are reasonable. This implies that only when the MAE of the Hamiltonian is sufficiently small can we ensure the corresponding orbital energy deviation is also small. In repeated experiments on the MD17 dataset, we found that when the Hamiltonian MAE of the model is at the same level, it is possible to obtain both a lower and a higher orbital energy MAE than the baseline. We believe this contributes to the performance difference observed between DEQHNet and the baseline model. This somewhat suggests that using existing Hamiltonian error measurements cannot definitively reflect the errors in the eigenvalues. This is a nontrivial, separate research question that we plan to investigate in the future.\n\nWe will elaborate on this point more clearly in the revised version of our paper to provide a comprehensive understanding of the model's performance.\n\n## Limitations:\nThank you for your insightful comments. Indeed, the DEQH model introduces self-consistency using DEQ, but it remains a supervised learning task that requires DFT data as labels. The statement \"DEQH model is a Hamiltonian solver\" refers to the DEQH model's ability to learn the iterative process of solving the Hamiltonian through DFT data, rather than it being a Partial Differential Equation (Schr\\\"odinger equation) solver. With this mechanism, DEQH is a better way for supervised learning of Hamiltonian in that it exploits more information in a model of limited size by passing through the model multiple times until convergence, and that the model learns the iteration map which is a physical mechanism that generalizes better.\n\nYour idea of bypassing DFT computations is intriguing and aligns with our thoughts. As we mentioned in Appendix A.11, some existing methods introduce self-consistency using unlabeled data. Our approach is entirely orthogonal to these methods. Therefore, combining these methods to achieve a trade-off between DFT computational cost and accuracy is a promising direction worth exploring.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We are grateful for the valuable feedback provided by all the reviewers. We provide a PDF document in the global rebuttal section, which includes a figure delineating the distinction between the off-the-shelf model used for Hamiltonian prediction and the DEQH model. We hope this visual aid will enable readers to gain a clearer and more intuitive understanding.

Meta-review: The paper introduces a novel neural architecture for quantum Hamiltonian prediction based on deep equilibrium models to model iterative algorithms. Benchmarks against competing methods show in general improvements in the MAE on Hamiltonian predictions. 
Some concerns highlighted by the reviews are around clarity of the presentation and the fact that better Hamiltonian prediction does not translate to better prediction of other molecular properties. The authors addressed these providing further experimental results during the discussion phase that helped understanding the performance of the model.
Decision: Accept (poster)