SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Adaptive Update Interval: Instead of updating  $\Sigma$  in every iteration, we control the update interval of  $\Sigma$  via a parameter  $I_{\Sigma}$  to improve efficiency. Since  $V$  is only updated slightly in each iteration step with a small learning rate  $\eta$ , it may not be necessary to do a fresh computation of  $\Sigma = F(V)$  in each time step, especially considering that the computation of  $U_{\mathrm{eff}}(\cdot)$  in Equation (9) can be costly. Moreover, we change  $I_{\Sigma}$  adaptively in the iteration to improve the efficiency by the following intuition: if there is a significant change between  $\Sigma_t$  and  $\Sigma_{t-1}$ , our dynamic PCA model is in an unstable state and we should apply online PCA method in a more responsive way to improve the convergence by setting a smaller  $I_{\Sigma}$ . However, if the gap is small, then our dynamic PCA model is very likely to be already close to a stable state, in which case the precision of PCA result is more dominant for obtaining a highly precise solution (especially for electronic structure calculation requiring error  $< 10^{-10}$ ), and a larger  $I_{\Sigma}$  is more reasonable. In this way, we set  $I_{\Sigma}$  to be inversely proportional to the difference between  $\Sigma_t$  and  $\Sigma_{t-1}$ , as  $I_{\Sigma}^{t} = \left\lceil \frac{1}{\Delta(\Sigma_{t}, \Sigma_{t-1})} \right\rceil$  in which  $\Delta(\Sigma_t, \Sigma_{t-1}) \geq 0$  is a properly scaled function evaluating the difference between successive  $\Sigma$ .
Adaptive PCA Mode Switching: Note that, the larger  $I_{\Sigma}$  is, the more costly in time between two updates of  $\Sigma$ , and the closer the result is to regular PCA on  $\mathcal{P}(x; \Sigma)$ . Thus a cut-off value  $T_{\mathrm{cut-off}}$  is set so that the PCA model will switch from online to regular when  $I_{\Sigma}^{t} > T_{\mathrm{cut-off}}$ , avoiding exhausted iteration on a single  $\Sigma$ . Moreover, this brings the preciseness needed for the final stage of the iteration when the error is small and convergence may not be a problem. Empirically, such a mode switching will introduce temporary disturbance for a few iterations, so we also set a small "tabu tenure"  $^2 T_{\mathrm{tabu}}$  prohibiting switching back to the original PCA mode in  $T_{\mathrm{tabu}}$  steps. Such switching between online and regular PCA can be triggered multiple times. A high amount of switches indicates that the convergence of the iteration may be hard, so  $T_{\mathrm{cut-off}}$  will increase by a small value  $T_{\mathrm{cut-off-inc}}$  after each switching to increase the proportion of online PCA to handle the convergence issue.
Additionally, we also applied the direct inversion of the iterative subspace (DIIS) method, which is empirically effective in traditional methods for electronic structure calculation. The basic idea is to update  $\Sigma_{t}$  at iteration  $t$  as a linear combination of matrices in previous  $T$  iterations  $\Sigma_{t - T},\dots ,\Sigma_{t - 1}$ , whose detail can be found in [28] and the appendix. In the experiments involving electronic structure calculation, we apply DIIS on all tested methods.
Some other numerical adaptations, including the application of momentum and sample-free update, are also leaved in the appendix. Summarizing all considerations above, we propose an algorithm for solving Equation (8), named "Adaptive Online SCF", shown in Algorithm 1
2This term is borrowed from Tabu Search [9].
Algorithm 1 Adaptive Online SCF for solving Equation (8) in electronic structure calculation
Input:  $H,S,U_{\mathrm{eff}}(\cdot)$  in Equation (8) and (9), learning rate  $\eta$  , difference evaluation function  $\Delta_1(\Sigma_t,\Sigma_{t - 1})$  for convergence criteria and  $\Delta_2(\Sigma_t,\Sigma_{t - 1})$  for the computation of  $I_{\Sigma}$  , cut-off threshold  $T_{\mathrm{cut - off}}$  and increment value  $T_{\mathrm{cut - off - inc}}$  tabu tenure  $T_{\mathrm{tabu}}$  , convergence threshold  $\epsilon$    
Output:  $V^{*}$  , the solution of Equation (8)   
1: Initialize  $V_0^\prime \in \mathbb{R}^{N\times k}$  randomly.   
2:  $\Sigma_0\gets H$    
3: Find  $X$  satisfying  $X^{\top}SX = I$    
4:  $s_0\gets$  Online   
5:  $t\gets 0,i\gets 0$    
6: while  $\Delta_1(\Sigma_t,\Sigma_{t - 1}) > \epsilon$  do   
7:  $V_{t}\leftarrow X V_{t}^{\prime}$    
8:  $\Sigma_t\gets \mathrm{DIIS}(\Sigma_{t - S},\dots ,\Sigma_{t - 1},F(V))$    
9:  $\Sigma_t^\prime \gets X^\top \Sigma_tX$    
10:  $I_{\Sigma}^{t} = \lceil 1 / \Delta_{2}(\Sigma_{t},\Sigma_{t - 1})\rceil$    
11: if  $I_{\Sigma}^{t}\leq T_{\mathrm{cut - off}}$  and not ( $s_t =$  Regular and  $i <   T_{\mathrm{tabu}}$  ) or ( $s_t =$  Online and  $i <   T_{\mathrm{tabu}}$  ) then   
12: for  $t^{\prime} = 0,1,2,\dots ,I_{\Sigma}^{t}$  do   
13: Use online PCA to update  $V_{t}^{\prime}$  towards "covariance matrix"  $-\Sigma_t^\prime$  , with momentum.   
14: end for   
15:  $s_t\gets$  Online   
16: else   
17: Use regular PCA to compute "principal component"  $V_{t}^{\prime}$  from "covariance matrix"  $-\Sigma_t^\prime$  .   
18:  $s_t\gets$  Regular   
19: end if   
20:  $i\gets i + 1$  if  $s_t = s_{t - 1}$  else 0   
21:  $T_{\mathrm{cut - off}}\gets T_{\mathrm{cut - off}}$  if  $s_t = s_{t - 1}$  else  $T_{\mathrm{cut - off}} + T_{\mathrm{cut - off - inc}}$    
22:  $t\gets t + 1$    
23: end while   
24:  $V^{*}\gets XV^{\prime}$
4.1 Experimental Verification of the Case Study
To verify the convergence analysis in Section 3.4, we conduct Monte Carlo experiments to solve Equation (5) for rotation matrix  $A$  with different rotation angle  $\theta$ . For each angle, we uniformly sample 1,000 initial vectors  $v_{0}$  on the unit sphere, perform the online PCA method and traditional fixed-point iteration (vanilla SCF) starting from these vectors to solve Equation (5), and use the proportion of converged vectors as an approximation of convergence probability. To be aligned with the next section, we also included the DIIS method described in the appendix that is extensively used in solving Equation (8).
The result is shown in Figure 5. While vanilla SCF (fixed-point iteration) cannot converge on any instance, the convergence probability of online PCA is aligned with the analytical result shown in Figure 4c. DIIS achieves around  $40\%$  convergence probability regardless of the rotation angle, which is significantly better than vanilla SCF but still fall behind for  $\theta < 3\pi /4$  compared with online PCA method. The result validates the convergence analysis in Section 3.4, showing the potentially high convergence capacity of online PCA method.
4.2 Convergence Capability Evaluation on Electron Structures
In this section, we perform extensive benchmarks on the QM9 dataset [31, 29], a diverse, large dataset to evaluate the capacity of converging Equation (8), the SCF equation in the scenario of electronic structure calculation. The QM9 dataset contains the atomic coordinates of 133,885 molecules in total, which is huge in size, thus we sampled  $1\%$  of the dataset (1,338 molecules) in a purely random manner for our evaluation. The effective potential matrix function  $U_{\mathrm{eff}}(\cdot)$  in Equation (9) is based on Hartree-Fock theory and Density Functional Theory (DFT) with B3LYP exchange-correlation functional, provided by PySCF [32]. We evaluated the methods on the two theories separately.
(a) Convergence ratio
(b) An example of convergence trajectories
Figure 5: Performance evaluation of solving Equation (5) whose analytical solutions are available. (a) While vanilla SCF (fixed-point iteration) cannot converge on any instance, the convergence probability of online PCA is aligned with the analytical result shown in Figure 4c. (b) An example of convergence trajectories, traditional SCF (fixed-point iteration) fails to converge (cycling around the solutions) while our Online PCA method converges to the solutions (red and blue arrow) smoothly.
Methods	Hartree-Fock	DFT with B3LYP
#(Nonconverged molecules)	Average #(iterations)	#(Nonconverged molecules)	Average #(iterations)
Regular SCF	124	(9.27%)	25.49	407	(30.42%)	21.09
Full Online SCF	13	(0.97%)	584.68	217	(16.22%)	1835.24
Adaptive Online SCF	0	(0%)	42.97	0	(0%)	60.58
Table 1: Results on 1,338 randomly sampled molecules in QM9 dataset. All methods are initialized with core Hamiltonian and accelerated by DIIS. Average #(iterations) is for converged molecules only.
Standard 6-31G basis set [5] are applied for the computation of all molecules. All tested methods are initialized with core Hamiltonian and accelerated by DIIS. Three methods are evaluated as follows:
- Regular SCF: The default method in PySCF with DIIS enabled, as similar to most of the quantum chemistry software.
- Full Online SCF: Algorithm 1 without the adaptive mode switching mechanism. Online SCF is applied throughout the whole iteration process, with a learning rate of  $10^{-2}$ . To avoid the explosion of update interval  $I_{\Sigma}^{t}$  when approaching to convergence, we simply set an upper limit of 10,000 for  $I_{\Sigma}^{t}$ .
- Adaptive Online SCF: Algorithm 1 including the adaptive mode switching mechanism. Regular SCF is allowed to kick in when the iteration process is close to convergence.  $T_{\mathrm{cut-off}}$  and  $T_{\mathrm{cut-off-inc}}$  are set to be 100 and 10 respectively.  $T_{\mathrm{tabu}}$  is set to be 10.
The result is shown in Table 1. Compared with regular SCF approach, full online SCF method significantly reduced the number of nonconverged molecules in both Hartree-Fock and DFT scenarios, demonstrating its high convergence capacity in solving Equation (8). However, the gradient-like update rule of online methods results in comparatively low precision. This not only increases the number of required iterations significantly, but also restricts it from achieving higher convergence capacity under strict convergence criteria. The adaptive mode switching mechanism successfully resolved the issue. By allowing regular SCF to kick in at a later stage, with the flexibility to return back to online mode when oscillation occurs, adaptive online SCF achieves converged solution for all test molecules, with a moderate increase of average iteration number.
The behavior of adaptive online SCF is shown in Figure 6. While most of the molecules get converged with only one mode transition in Figure 6a, there are also a few "hard cases" like Figure 6b that require multiple mode transitions between online and regular SCF. The detailed statistics is shown in Table 2. The capability of mode switching is essential for the convergence capacity, since it provides unlimited chances of trials to reach a converged trajectory, which is lacking in regular SCF methods with only a few choices of starting point to select.
(a) Converge with one mode transition.
(b) Converge with multiple mode transitions.
Figure 6: Examples of converge curves for adaptive online SCF.
#(mode transition)	1	2	3	4	5	6	7	8	>8
Hartree-Fock	1143	179	13	2	0	0	1	0	0
DFT with B3LYP	1025	189	60	27	18	6	6	3	4
Table 2: Mode transition statistics of adaptive online SCF on sampled QM9 dataset. While all molecules finally converged on both Hartree-Fock and DFT scenarios and most of the them only require one mode transition from online to regular, the distribution of mode transition for DFT is more long-tailed (the 4 molecules with  $\# (\text{mode transition}) > 8$  have 12, 13, 15 and 32 transitions respectively).
In this work, we take the first steps in devising PCA-based algorithms for converging non-linear equations, and demonstrate successful applications in SCFs. This work contributes to both the field of computational science and machine learning as follows:
- For computational science, this work presents a new algorithm to converge SCF equations in electronic structure calculation with high successful rate, especially without any heuristics based on prior quantum mechanism knowledge to bootstrap the solving stage.
- For machine learning, this work explores a brand new area of "self-consistent" eigenvalue problems, especially SCF equations, for online PCA methods such as Oja's algorithm and EigenGame, which are previously regarded as specialized methods for  $k$ -PCA. While such methods can properly handle data stochasticity, this work shows that they are also capable of handling self-consistency, which leads to a potential of application in a broader field of scientific computing.
[1] Zeyuan Allen-Zhu and Yuanzhi Li. First efficient convergence for streaming k-pca: A global, gap-free, and near-optimal rate. In 2017 IEEE 58th Annual Symposium on Foundations of Computer Science (FOCS), pages 487-492, 2017.
[2] Stefan Banach. Sur les opérations dans les ensembles abstracts et leur application aux équations intégrales. Fundamenta Mathematicae, 3(1):133-181, 1922.
[3] Thomas D. Barrett, Aleksei Malyshev, and A. I. Lvovsky. Autoregressive neural-network wavefunctions for ab initio quantum chemistry. Nature Machine Intelligence, 4(4):351-358, April 2022. Number: 4 Publisher: Nature Publishing Group.
[4] Eric Cances, Mireille Defranceschi, Werner Kutzelnigg, Claude Le Bris, and Yvon Maday. Computational quantum chemistry: A primer. In Special Volume, Computational Chemistry, volume 10 of Handbook of Numerical Analysis, pages 3-270. Elsevier, 2003. ISSN: 1570-8659.
[5] R. Ditchfield, W. J. Hehre, and J. A. Pople. Self-consistent molecular-orbital methods. ix. an extended gaussian-type basis for molecular-orbital studies of organic molecules. The Journal of Chemical Physics, 54(2):724-728, 1971.
[6] Alejandro J. Garza and Gustavo E. Scuseria. Comparison of self-consistent field convergence acceleration techniques. The Journal of Chemical Physics, 137(5):054110, August 2012.
[26] Erkki Oja. Simplified neuron model as a principal component analyzer. Journal of mathematical biology, 15(3):267-273, 1982.
[27] David Pfau, James S. Spencer, Alexander G. D. G. Matthews, and W. M. C. Foulkes. Ab initio solution of the many-electron schrödinger equation with deep neural networks. Phys. Rev. Research, 2:033429, Sep 2020.
[28] Péter Pulay. Convergence acceleration of iterative sequences. the case of scf iteration. Chemical Physics Letters, 73(2):393-398, 1980.
[29] Raghunathan Ramakrishnan, Pavlo O Dral, Matthias Rupp, and O Anatole von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific Data, 1, 2014.
[30] C. C. J. Roothaan. New developments in molecular orbital theory. Rev. Mod. Phys., 23:69-89, Apr 1951.
[31] Lars Ruddigkeit, Ruud van Deursen, Lorenz C. Blum, and Jean-Louis Reymond. Enumeration of 166 billion organic small molecules in the chemical universe database gdb-17. Journal of Chemical Information and Modeling, 52(11):2864–2875, 2012. PMID: 23088335.
[32] Qiming Sun, Timothy C. Berkelbach, Nick S. Blunt, George H. Booth, Sheng Guo, Zhendong Li, Junzi Liu, James D. McClain, Elvira R. Sayfutyarova, Sandeep Sharma, Sebastian Wouters, and Garnet Kin-Lic Chan. Pyscf: the python-based simulations of chemistry framework. WIREs Computational Molecular Science, 8(1):e1340, 2018.
[33] Attila Szabo and Neil S Ostlund. Modern quantum chemistry: introduction to advanced electronic structure theory. Dover Publications, Inc., 1996.
[34] Cheng Tang. Exponentially convergent stochastic k-pca without variance reduction. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[35] JH Van Lenthe, R Zwaans, Huub JJ Van Dam, and MF Guest. Starting scf calculations by superposition of atomic densities. Journal of computational chemistry, 27(8):926-932, 2006.