SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xihan Li $^{1}$ , Xiang Chen $^{2}$ , Rasul Tutunov $^{3}$ , Haitham Bou Ammar $^{3}$ , Lei Wang $^{4}$ , Jun Wang $^{1}$ $^{1}$  University College London  $^{2}$  Huawei Noah's Ark Lab  $^{3}$  Huawei R&D U.K.
$^{4}$  Institute of Physics, Chinese Academy of Sciences {xihan.li, jun.wang}@cs.ucl.ac.uk {xiangchen.ai, rasul.tutunov, haitham.ammar}@huawei.com wanglei@iphy.ac.cn
Self-consistent Field (SCF) equation is a type of nonlinear eigenvalue problem in which the matrix to be eigen-decomposed is a function of its own eigenvectors. It is of great significance in computational science for its connection to the Schrödinger equation. Traditional fixed-point iteration methods for solving such equations suffer from non-convergence issues. In this work, we present a novel perspective on such SCF equations as a principal component analysis (PCA) for non-stationary time series, in which a distribution and its own top principal components are mutually updated over time, and the equilibrium state of the model corresponds to the solution of the SCF equations. By the new perspective, online PCA techniques are able to engage in so as to enhance the convergence of the model towards the equilibrium state, acting as a new set of tools for converging the SCF equations. With several numerical adaptations, we then develop a new algorithm for converging the SCF equation, and demonstrated its high convergence capacity with experiments on both synthesized and real electronic structure scenarios.
In this work, we are concerned with the convergence issue of solving the following type of nonlinear eigenvalue problem:
$$
F (v) v = \lambda v \tag {1}
$$
in which  $F(v)$  is a given mapping from an  $N$ -dimensional unit vector to an  $N \times N$  positive semi-definite matrix,  $\lambda$  is the largest eigenvalue of  $F(v)$ , and  $v$  is the normalized eigenvector of  $F(v)$  corresponding to  $\lambda$ . The core challenge of this problem is the involvement of self-consistency, that is, the form of the eigenvalue equation (i.e., the matrix  $F(v)$  to be decomposed) is defined by its final solution  $v$ , but the solution itself is not directly accessible unless the form of the equation is given, which is a paradox. Such type of nonlinear eigenvalue problem is especially concerned in computational science, since some variations of Equation (1) like Hartree-Fock equations [10, 11] and Kohn-Sham equations [15, 19] are the foundation of electronic structure calculation by approximating the Schrödinger equation, in which  $F(v)$  corresponds to the Fock operator approximating the Hamiltonian operator of a quantum system, and the eigenvector  $v$  corresponds to the coefficients of an orbital wave function under certain basis [33, 25]. Such equations are usually called "self-consistent field (SCF) equations".
However, like many other types of nonlinear equations, no existing numerical algorithms can solve SCF equations with theoretical convergence guarantee, while many works are committed to improve the practical "successful rate" for the solution to be converged. To be more specific, the iterative
method to solve SCF equations is generally referred to as self-consistent field (SCF) method [30, 23], whose basic idea is to regard the SCF equation as a fixed-point equation  $v = f(v)$  so as to perform fixed-point iteration. For Equation (1), we can rewrite it as  $v = \operatorname{Eig}(F(v))$  in which  $\operatorname{Eig}(\cdot)$  returns the eigenvector of a matrix corresponding to its largest eigenvalue. Then the fixed-point iteration is performed by generating an initial solution  $v_{0}$  and performing  $v_{t+1} = \operatorname{Eig}(F(v_{t}))$  iteratively until convergence. Note that there is no theoretical guarantee that the above iteration step leads to a converged solution<sup>1</sup>. In practice, vanilla SCF method also easily fails in real electronic structure calculation under Hartree-Fock or Kohn-Sham equations, acting as oscillating between two or more different states that are not solutions of the equations [4]. To mitigate the convergence issue of SCF method, existing works mainly follow two directions: one is to generate better-quality initial solutions  $v_{0}$  in a semi-empirical way [14, 35, 22], another one is to mix  $F(v_{t})$  with those in previous iterations  $F(v_{t-1}), F(v_{t-2}), \dots$  to stabilize the iteration process [28, 21, 16, 6].
In this work, we propose a different direction to converge SCF equations. We have an insight that the essence of SCF equation is a special type of eigen-decomposition where the matrix to be decomposed is not determined during the decomposition. In this way, it shares a key similarity with principal component analysis (PCA) in a non-stationary time-series environment, as PCA is also rooted on eigen-decomposition, and the covariance matrix of the data distribution is not determined as it may change over time. By transiting to the PCA perspective, it is possible to apply a series of online PCA techniques that are successfully developed to handle PCA in non-stationary time-series (or streaming) environments. They are able to adapt to new patterns in the data stream dynamically by incremental updates of the principal component. Connecting it with SCF equations, such an incremental feature is potentially helpful in mitigating the convergence issue, especially by preventing long-range oscillation between successive iterations. Motivated by these insights, we propose a dynamic PCA model with an auto-encoder structure, whose equilibrium state is the solution of Equation (1). In this model, we view the eigenvector  $v$  in the SCF equation as a principal component for a certain data distribution  $\mathcal{P}(x; \Sigma)$  in PCA, and interpret  $F(v)$  in the SCF equation as a "reconstruction function" that convert a vector  $v$  to the covariance matrix  $\Sigma$  of a certain distribution. Then we utilize online learning techniques to lead the dynamic PCA model towards equilibrium state.
For some important real-world applications of SCF equations, such as Hartree-Fock and Kohn-Shan equations for electronic struction calculation, we also show that our PCA-based model can be adapted to these applications, and proposed several numerical adaptations to converge these more complicated equations. Particularly, we proposed an adaptive mechanism that allows the iteration to switch between online mode for convergence and regular mode for accuracy, which provides unlimited chances of trials to reach a converged trajectory, which is lacking in standard SCF methods when stuck in oscillation.
Experimental result on synthesis and real scenarios shows that our proposed approach largely reduce the occurrence of oscillation, leading to a significant improvement of convergence performance. In this way, our work expands the reach of online PCA methods into handling self-consistency. In summary, we make the following contributions:
- A novel formulation of the SCF equation as finding the equilibrium state of a dynamic PCA model for non-stationary time series.
- A new application of online learning techniques to the proposed model so as to enhance its convergence to the equilibrium state and avoid oscillation, improving the convergence performance of SCF equations solving in a generic way.
- A new algorithm based on online PCA with several numerical adaptations, which is capable of converging the SCF equation in real-world electronic structure calculation with high successful rate.
- Extensive experiments on a synthetic problem and real datasets for electronic structure calculation, demonstrating the high capacity of the proposed algorithm in converging the SCF equation.
To have an insight, consider that the mapping  $\operatorname{Eig}(F(\cdot)) : \mathbb{R}^N \to \mathbb{R}^N$  is generally not a contraction mapping so that the Banach fixed-point theorem [2] does not work here.
With the prosperity of deep learning and differentiable optimization in recent years, there are works combining machine learning with computational science and electronic structure calculation, including deep-learning-aided wave function representation of quantum Monte Carlo methods such as FermiNet [27], PauliNet [13] and [3] and neural representation of the exchange-correlation functional in density functional theory [24, 18]. However, these works stay within traditional formalism of the problem, focusing more on improving simulation accuracy towards physical reality by using neural networks as better functional approximators, while our work's focus is very different, stressing on a machine learning oriented formalism of the SCF equation, which has not been explored in the literature to the best of our knowledge.
To converge SCF equations, existing works mainly follow two directions. One is to generate better-quality initial solutions  $v_{0}$  [14, 35, 22]. However, these methods are generally semi-empirical as they require specific assumption of  $F$ , and leverage domain knowledge (e.g., quantum mechanism) for the initialization. another one is to mix  $F(v_{t})$  with those in previous iterations  $F(v_{t - 1}), F(v_{t - 2}), \dots$  to stabilize the iteration process [28, 21, 16, 6]. While these methods perform well on converging SCF equations efficiently, they can still be stuck in indefinite oscillating between non-solution states without the chance of escaping from it.
Principal component analysis (PCA) is a fundamental, well-studied tool used to for data analysis and compression [17]. The principal components, which are "representative directions" of the data distribution that preserve the data's variation, can be computed by eigen-decomposition of the data covariance matrix. However, when the data are formed as an online, non-stationary stream whose distribution (or more specifically, the covariance matrix) may shift over time, specialized online learning techniques are required to estimate the top- $k$  principal components in a real-time manner, and dynamically adapt to new patterns in the data stream. Starting from Hebb's rule in neuroscience [12], a series of works [26, 34, 1, 7] focus on this direction named Online  $k$ -PCA.
3 Solving SCF Equations with Online PCA
To mitigate the convergence issue in the solving of the SCF equation (1), we connect it with PCA by proposing a new PCA model for non-stationary time series, showing that the solving of the SCF equation is equivalent to finding the equilibrium state of the proposed PCA model. In this way, online learning techniques for PCA can be exploited to accelerate the convergence. We also propose some numerical adaptations so as to solve real-world SCF equations in electronic structure computation.
3.1 The Connection between SCF Equations and PCA
Figure 1: SCF equations as finding a distribution that is invariant before and after being processed by an auto-encoder structure involving PCA. An input distribution  $\mathcal{P}(\pmb{x};\pmb{\Sigma})$  is compressed yielding a set of principal components  $\pmb{v}_{1:n}$ . Those are then used to produce a reconstructed distribution  $\mathcal{P}(\pmb{x};\pmb{\Sigma}')$ , where ideally we would like  $\pmb{\Sigma} = \pmb{\Sigma}'$ .
To connect the SCF equation with PCA, notice that Equation (1) can also be stated as
$$
v = \operatorname {E i g} (F (v)) \tag {2}
$$
$$
\Sigma = F (\operatorname {E i g} (\Sigma)) \tag {3}
$$
Figure 2: A diagram of the dynamic PCA model, in which a distribution  $\mathcal{P}(\pmb{x};\pmb{\Sigma})$  and its top principal component  $\pmb{v}_{1:n}$  are mutually updated over time. Importantly, we notice that we can regard the approach as applying PCA on non-stationary distributions that evolve over time. Given that such a formalism is novel and is an outcome of SCFs, the machine learning literature is yet to devise effective solutions. In this work, we take the first steps in devising such algorithms and demonstrate successful applications in SCFs.
where  $\operatorname{Eig}(\cdot)$  is the eigenvector of a matrix corresponding to its largest eigenvalue, and  $v = \operatorname{Eig}(\Sigma)$ . Here, if we consider the function  $\Sigma = F(v)$  as a "reconstruction function" that converts a vector  $v$  to the covariance matrix  $\Sigma$  of a certain data distribution  $\mathcal{P}(x; \Sigma)$ , then the function  $v = \operatorname{Eig}(\Sigma)$  is equivalent to a "compression function" that finds the top principal component  $v$  of this distribution  $\mathcal{P}(x; \Sigma)$  with PCA, since the top principal component is exactly the eigenvector of the distribution's covariance matrix corresponding to the largest eigenvalue. From this perspective, the solving of Equation (3) (and equivalently (2)) can be seen as finding a distribution  $\mathcal{P}(x; \Sigma)$  parameterized by the covariance matrix  $\Sigma$  that is invariant before and after being processed by the following two stages:
- Compression (PCA): perform PCA on the distribution  $\mathcal{P}(x; \Sigma)$  so as to obtain its top principal component  $v$ , which is the representative direction of the distribution.
- Reconstruction  $(F(v))$ : process  $v$  with the given reconstruction function  $\Sigma' = F(v)$  so as to obtain the reconstructed distribution  $\mathcal{P}(x; \Sigma')$ .
which is shown in Figure 1. While its architecture seems similar to the autoencoder [20], the encoder (PCA) and decoder (reconstruction function  $F(v)$ ) here are both fixed without any adjustable parameters. Instead of training the encoder and decoder, here we aim to find an input distribution  $\mathcal{P}(x; \Sigma)$  which remains invariant after been "coded" and "decoded".
3.2 A New PCA Model for Non-stationary Time Series
For solving of the SCF equation (2), traditionally we perform fixed-point iteration by letting the fixed-point mapping  $f(v) = \operatorname{Eig}(F(v))$  and performing  $v_{t+1} = f(v_t)$  iteratively until convergence. Here, by replacing  $f$  with the reconstruction and compression stages mentioned above, we obtain an equivalent form of the fixed-point iteration, which is presented as a dynamic model that a distribution  $\mathcal{P}(x; \Sigma)$  and its top principal component  $v$  are mutually updated over time, shown in Figure 2. The evolution process of the dynamic PCA model is as follows:
Given initial top principal component  $v_{0}$ , reconstruction function  $F(v)$  For  $t = 1,2,\dots$  until converge  $(\| v_{t} - v_{t - 1}\| < \epsilon)$
- Reconstruct the distribution  $\mathcal{P}(x; \Sigma_t)$  by  $\Sigma_t = F(v_{t-1})$ .
- Perform PCA on  $X_{t} \sim \mathcal{P}(x; \Sigma_{t})$  and obtain the top principal component  $v_{t} \gets \mathrm{PCA}(X_{t})$ .
In this model, the top principal component is updated by performing PCA on the current distribution, and the distribution is updated by performing reconstruction on the current top principal component. As it is equivalent to the fixed-point iteration, the equilibrium state of the new dynamic model is also the fixed point of Equation (2), which corresponds to the solution of Equation (1).
An important feature of the dynamic PCA model is that, it can be regarded as applying PCA on a non-stationary distribution over time, as the distribution that is processed by PCA is subject to change
Figure 3: Convergence comparison between vanilla fixed-point iteration and Online PCA given  $F(v) = \left[ \begin{array}{cc} v_2^2 & v_1v_2 \\ v_1v_2 & v_1^2 \end{array} \right]$ , if we set the initial solution  $v_0$  as  $[1/2, \sqrt{3}/2]^{\top}$ . The principal component at time step 0 is set to be  $[1/2, \sqrt{3}/2]^{\top}$  for both models. While the vanilla fixed-point iteration oscillates between  $[1/2, \sqrt{3}/2]^{\top}$  and  $[\sqrt{3}/2, 1/2]^{\top}$  along with the time step, the dynamic PCA model proposed in Section 3.2 converges to the ground-truth solution  $[\sqrt{2}/2, \sqrt{2}/2]^{\top}$  steadily after applying online PCA technique.
over time in the evolving process. This new perspective enables a variety of PCA techniques for non-stationary environments to be applied on the solving of the problem, whose details are discussed below.
3.3 Online PCA for Converging SCF Equations
As we have discussed in the introduction, vanilla fixed-point iteration method has serious convergence issues. acting as oscillating between two or more different points, neither of which are the solution of Equation (2). For example, given  $F(v) = \left[ \begin{array}{cc} v_2^2 & v_1v_2 \\ v_1v_2 & v_1^2 \end{array} \right]$ , if we set the initial solution  $v_0$  as  $[1/2,\sqrt{3}/2]^{\top}$  and perform fixed-point iteration  $v_k = \operatorname{Eig}(F(v_{k-1}))$ , we will find it oscillating between  $[1/2,\sqrt{3}/2]^{\top}$  and  $[\sqrt{3}/2,1/2]^{\top}$  along with the time step. Neither of them are close to the analytical solution  $[\sqrt{2}/2,\sqrt{2}/2]^{\top}$ . However, the new PCA-based perspective allows us to apply online learning techniques to mitigate this issue. In online PCA, the principal components are usually updated in an incremental manner so as to adapt to new patterns in the data distribution. The incremental property is especially appealing to us, since the oscillation of fixed-point iteration behaves as infinite "jumps" between different states, and incremental updates can reduce such jumps by adding soft restrictions to the difference of principal components between successive time steps. After applying online learning techniques, the evolution process of the dynamic PCA model is as follows:
Given initial top principal component  $v_{0}$ , reconstruction function  $F(v)$
For  $t = 1,2,\dots$  until converge  $(\| v_{t} - v_{t - 1}\| <  \epsilon)$
- Reconstruct the distribution  $\mathcal{P}(x; \Sigma_t)$  by  $\Sigma_t = F(v_{t-1})$ .
- Sample  $x_{t}$  from  $\mathcal{P}(x;\Sigma_t)$
- Update  $v$  by online PCA with  $x_{t}$
and an illustrative example is shown in Figure 3.
3.4 A Case Study for the Convergence Behavior of Online PCA
For the iterative methods solving Equation (1), while the convergence analysis is generally intractable due to the arbitrariness of  $F(v)$  (usually nonlinear by involving  $vv^{\top}$ ), here we provide a case study for a specific form of  $F(v)$  whose analytical ground truth solutions are available. While vanilla fixed-point iteration method doesn't work in this case, we can derive and visualize the convergence behavior of our proposed online PCA method in an analytical way. Here we let
$$
F (v) = \left(A v\right) \left(A v\right) ^ {\top} = A v v ^ {\top} A ^ {\top} \tag {4}
$$
in which  $A$  is an orthogonal matrix. Substitute Equation (4) into (1) and we obtain
$$
\left(A v v ^ {\top} A ^ {\top}\right) v = \lambda v \tag {5}
$$
The analytical solution of Equation (5) is the normalized eigenvector of  $A$  corresponding to the eigenvalue 1, since the largest eigenvalue of matrix  $A v v^{\top} A^{\top}$  is 1 with corresponding eigenvector
(a) Updating  $v$  with online PCA
(c) Convergence probability w.r.t.  $\theta$
Figure 4: Convergence analysis of our proposed online PCA method for solving Equation (5). Note that vanilla fixed-point iteration can never converge in this case.
(b) Convergence dynamics for  $\theta \in (\frac{\pi}{2},\pi)$
$Av$  (i.e.,  $\operatorname{Eig}(A vv^\top A^\top) = Av$ ), reducing Equation (5) to a standard eigenvalue problem  $v = Av$ . However, The vanilla fixed-point iteration method cannot converge when solving Equation (5). Note that the mapping in each iteration will be reduced to  $v_{t+1} = Av_t$ , which seems very similar to the power method on  $A$  for finding the eigenvector with the largest eigenvalue, but with convergence ratio  $\left|\frac{\lambda_2}{\lambda_1}\right| = 1$  (i.e., cannot converge) since all the absolute value of  $A$ 's eigenvalues are 1 as  $A$  is an orthogonal matrix.
Now we turn to the convergence analysis of our proposed online PCA method for this case. For the sake of visualization, we further restrict  $A$  to be three-dimensional as a rotation transformation matrix around the  $z$ -axis with angle  $\theta$ , that is
$$
A = \left[ \begin{array}{c c c} \cos \theta & - \sin \theta & 0 \\ \sin \theta & \cos \theta & 0 \\ 0 & 0 & 1 \end{array} \right] \tag {6}
$$
so that the solution of Equation (5) will be  $v^{*} = [0,0,\pm 1]^{\top}$ , the two unit vectors on the  $z$ -axis. Then, we let the initial top principal component  $v_{0}$  to be a unit vector on the  $yz$  plane
$$
v _ {0} = \left[ 0, \cos \varphi_ {0}, \sin \varphi_ {0} \right] ^ {\top}, \varphi_ {0} \in \left[ 0, \frac {\pi}{2} \right] \tag {7}
$$
where  $\varphi_0$  is the initial angle between the vector and the  $xy$  plane.
We analytically derived the convergence behavior of the online PCA method described in Section 3.3 for solving Equation (5), whose detail is lefted in the appendix. The result is as follows: for  $\theta \in (0, \frac{\pi}{2})$ , the online PCA method is guaranteed to converge to the ground-truth solution on  $z$ -axis. However, for  $\theta \in [\frac{\pi}{2}, \pi)$ , there is a phase transition of convergence behavior w.r.t. the initial angle  $\varphi_0$ , shown in Figure 4b. While the online PCA method stays converged for  $\varphi_0 > \arccos \sqrt{x_1(\theta)}$ , it will fail to converge otherwise. Assuming the initial vector is sampled uniformly on the unit sphere, the probability of convergence for the online PCA method w.r.t. the rotation angle  $\theta$  is shown in Figure 4c.
3.5 Numerical Adaptations for Converging Real-world SCF Equations
Now we investigate a more complicated and applicative scenario in electronic structure calculation, which is usually known as Hartree-Fock equations or Kohn-Sham equations. The form of such SCF equations [33] is
$$
F (V) V = S V \Lambda , \tag {8}
$$
-  $F(V)$ : an  $N \times N$  real symmetric matrix to be decomposed, as a given function of  $V$ . It is defined as follows:
$$
F (V) \stackrel {\text {d e f}} {=} H + U _ {\text {e f f}} \left(2 V V ^ {\top}\right), \tag {9}
$$
in which  $H$  is an  $N\times N$  real symmetric matrix and  $U_{\mathrm{eff}}(\cdot)$  is an  $N\times N$  real symmetric matrix as a function of  $2VV^{\top}$ . Both of them are given in the equation.
-  $S$ : an  $N \times N$  positive semi-definite matrix, which is a constant input in the problem.
-  $\Lambda$ :  $\Lambda = \mathrm{diag}(\lambda_1, \dots, \lambda_k)$  is a  $k \times k$  diagonal matrix containing the top- $k$  smallest eigenvalues.
-  $V\colon V = [v_{1},\dots ,v_{k}]$  is an  $N\times k$  matrix containing  $k$  normalized column eigenvectors corresponding to the top- $k$  smallest eigenvalues.
Our proposed model can be adapted to handle it as follows
1. While  $F(v)$  is not guaranteed to be positive semi-definite, note that we can always make an "eigenvalue shift" towards  $F(v)$  by adding it with  $\sigma I$  ( $\sigma$  is a moderately large number) to make it positive semi-definite, without changing its eigenvectors (i.e., the solution of Equation (8)).
2. For  $S \neq I$ , Equation (1) becomes  $F(v)v = \lambda Sv$  which is a generalized eigen-decomposition problem[8]. We can then perform standard orthogonalization technique that transforms the generalized eigenvalue problem into a standard one [33].
3. To tackle the smallest eigenvalues instead of the largest ones, we can simply add a minus sign to  $F(v)$ , so that the order of its eigenvalues will be reversed without changing its eigenvectors.
4. For  $k > 1$ , our proposed model can be trivially extended to  $k > 1$  cases by retaining  $k$  top principal components rather than one.
And we name the adapted model as "Online SCF". Then, we introduce some practical improvements that can further enhance the efficiency and convergence capability of our proposed model for solving Equation (8):