Title: Invariant subspaces and PCA in nearly matrix multiplication time
Venue: NeurIPS 2024
Accepted: yes
Keywords: Invariant subspace

Generalized eigenvalue problem

PCA

Spectral projector

Spectral gap

Matrix multiplication

Bit complexity
Abstract: Approximating invariant subspaces of generalized eigenvalue problems (GEPs) is a fundamental computational problem at the core of machine learning and scientific computing. It is, for example, the root of Principal Component Analysis (PCA) for dimensionality reduction, data visualization, and noise filtering, and of Density Functional Theory (DFT), arguably the most popular method to calculate the electronic structure of materials. 
Given Hermitian $H,S\in\mathbb{C}^{n\times n}$, where $S$ is positive-definite,  let $\Pi_k$ be the true spectral projector on the invariant subspace that is associated with the $k$ smallest (or largest) eigenvalues of the GEP $HC=SC\Lambda$, for some $k\in[n]$. 
We show that we can compute a matrix $\widetilde\Pi_k$ such that $\lVert\Pi_k-\widetilde\Pi_k\rVert_2\leq \epsilon$, in $O\left( n^{\omega+\eta}\mathrm{polylog}(n,\epsilon^{-1},\kappa(S),\mathrm{gap}_k^{-1}) \right)$ bit operations in the floating point model, for some $\epsilon\in(0,1)$, with probability $1-1/n$. Here, $\eta>0$ is arbitrarily small, $\omega\lesssim 2.372$ is the matrix multiplication exponent, $\kappa(S)=\lVert S\rVert_2\lVert S^{-1}\rVert_2$, and $\mathrm{gap}_k$ is the gap between eigenvalues $k$ and $k+1$. 
To achieve such provable "forward-error" guarantees, our methods rely on a new $O(n^{\omega+\eta})$ stability analysis for the Cholesky factorization, and a smoothed analysis for computing spectral gaps, which can be of independent interest.
Ultimately, we obtain new matrix multiplication-type bit complexity upper bounds for PCA problems, including classical PCA and (randomized) low-rank approximation.

Invariant subspaces and PCA in nearly matrix multiplication time

Abstract

Approximating invariant subspaces of generalized eigenvalue problems (GEPs) is a fundamental computational problem at the core of machine learning and scientific computing. It is, for example, the root of Principal Component Analysis (PCA) for dimensionality reduction, data visualization, and noise filtering, and of Density Functional Theory (DFT), arguably the most popular method to calculate the electronic structure of materials. Given Hermitian H , S ∈ C n × n , where S is positive-definite, let Π k be the true spectral projector on the invariant subspace that is associated with the k smallest (or largest) eigenvalues of the GEP HC = SCΛ , for some k ∈ [ n ] . We show that we can compute a matrix ˜ Π k such that ∥ Π k -˜ Π k ∥ 2 ≤ ϵ , in O ( n ω + η polylog( n, ϵ -1 , κ ( S ) , gap -1 k ) ) bit operations in the floating point model, for some ϵ ∈ (0 , 1) , with probability 1 -1 /n . Here, η > 0 is arbitrarily small, ω ≲ 2 . 372 is the matrix multiplication exponent, κ ( S ) = ∥ S ∥ 2 ∥ S -1 ∥ 2 , and gap k is the gap between eigenvalues k and k +1 . To achieve such provable 'forward-error' guarantees, our methods rely on a new O ( n ω + η ) stability analysis for the Cholesky factorization, and a smoothed analysis for computing spectral gaps, which can be of independent interest. Ultimately, we obtain new matrix multiplication-type bit complexity upper bounds for PCA problems, including classical PCA and (randomized) low-rank approximation.

1 Introduction

Generalized eigenvalue problems (GEPs) arise naturally in a plethora of applications in machine learning, scientific computing, and engineering. Given a pair of matrices H and S , often referred to as a matrix pencil , the problem of interest has the following form

where C and Λ are the unknown eigenvector and eigenvalue matrices, respectively. Of particular importance are the so-called 'Hermitian definite' or simply 'definite' GEPs/pencils, in which case H is Hermitian and S is Hermitian and positive-definite. In many important applications, the quantity of interest is an (arbitrarily large) subset of eigenvectors, defining an invariant subspace , rather than the entire C and Λ solutions of the GEP.

In data science and machine learning, invariant subspaces play a central role in many problems, including Spectral Clustering [111, 124], Language Models [73], Image Processing [118, 11], Recommendation Systems [46], Principal Components Analysis (PCA) [45, 79, 122], Support Vector Machines [99], and many others [50, 35, 96, 12]. We particularly focus on PCA applications, which can take the form of a GEP as in Eq. (1) where H is the sample covariance and S the identity. In more advanced settings, H and S can be defined over a kernel [122, 20]; See Section 4 and Appendix G for more details. Another closely related application comes from Density Functional

Theory [88] (DFT), which is not a machine learning problem per se, but it is probably the most commonly used method (it was awarded the Nobel prize in Chemistry in 1998) to compute the electronic and structural properties of materials. In this case, H is the Hamiltonian and S the overlap matrix (cf. Appendix F). The spectral projector on the invariant subspace corresponding to the smallest generalized eigenvalues (occupied energies) directly provides the density matrix and the electron density. Obtaining them often presents a challenge from the computational point of view.

1.1 Problem definition

The main focus of this work is the computation of spectral projectors on invariant subspaces that are associated with a subset of the spectrum of Hermitian definite GEPs. As the Abel-Ruffini theorem excludes exact computation, even in exact arithmetic, we seek for approximate computations, as described in the following Problem 1.1.

Problem 1.1 (Spectral projector) . Given a Hermitian definite GEP HC = SCΛ of size n , an integer 1 ≤ k ≤ n -1 , and accuracy ϵ ∈ (0 , 1) , compute a matrix ˜ Π k ∈ C n × n such that

where Π k is the true spectral projector on the invariant subspace associated with the k smallest or largest eigenvalues.

Before proposing algorithms to solve Problem 1.1, we first make some clarifications and define useful concepts.

Type of approximation: The approximation of the form of Equation (2) is commonly called a 'forward approximation' or 'forward error' in numerical analysis. It quantifies the distance between the true solution of the problem and the one returned by an algorithm. It is a stronger and harder to achieve notion of approximation than the related 'backward error.' For details see Appendix A.1.

Model of computation: While many finite precision models of computation exist in the literature, all algorithms in this work are analyzed in the floating point model of computation, which is also the prominent model implemented in existing computers. Each real number α is rounded to a floating point number fl ( α ) = (1 + θ ) α , where θ ∈ C satisfies | θ | ≤ u ∈ R > 0 . The machine precision u bounds also the errors introduced by arithmetic operations { + , -, × , / } , and the expression log(1 / u ) gives the number of bits required to achieve the desired precision. More details can be found in Appendix A.2.

Bit complexity: The complexity of numerical algorithms is often measured in terms of the arithmetic operations executed, commonly referred to as arithmetic complexity . A more realistic notion is the bit complexity , which bounds the number of boolean operations. In the floating point model, it is straightforward to translate the arithmetic to the bit complexity if we have an upper bound on the number of bits. For instance, arithmetic operations on b bits can be typically carried out in O ( b 2 ) bit operations, or even faster by using more advanced algorithms [123, 55, 68].

Matrix multiplication time: In two seminal works [39, 40], it was demonstrated that matrix multiplication and other fundamental problems in Numerical Linear Algebra can be solved in the floating point model with nearly O ( n ω + η ) bit-complexity (up to polylogarithmic factors), where η is an arbitrarily small positive number and ω is the matrix multiplication exponent, to-date bounded by ω ≲ 2 . 372 [47, 141, 6]. Hereafter, we will use the notation T MM ( n ) = n ω + η .

1.2 Existing algorithms

Here we give a brief overview of existing algorithms. We refer to Appendix A.7 for more details. GEPs in general can be solved using classic eigensolvers and related techniques in ˜ O ( n 3 ) floating operations, e.g., by reducing the matrix (or pencil) to tridiagonal form with similarity transformations and applying the shifted QR algorithm on the tridiagonal matrix, or by using a divide-andconquer method (see [37, 109, 75, 8, 9, 64, 42, 10, 113] and references therein). Significant progresses beyond the ˜ O ( n 3 ) bit complexity barrier have been made [40, 39, 15, 95, 25, 41, 107, 121].

Regarding the computation of eigenvalues, two notable examples are the ˜ O ( T MM ( n )) algorithm of [95] for the largest eigenvalye, and the ˜ O ( n 2 ) algorithm of [107] for the spectral norm.

The first to have addressed the problem of computing invariant subspaces in nearly O ( T MM ( n )) in floating point is [39] (see also [14, 13]). The authors described an iterative algorithm for the Schur decomposition, and showed that each individual step is numerically stable, and it takes O ( T MM ( n ) operations. An end-to-end bound on the number of iterations to achieve a backward approximate solution was left open. More recently, the seminal work of [15] extended the analysis to obtain an end-to-end ˜ O ( T MM ( n )) complexity to approximately diagonalize a matrix, and [41, 121] provided a rigorous analysis for the generalized eigenproblem case. In Corollary 1.7 and Proposition 1.1 of [15], it was also outlined how to translate the backward diagonalization error to a forward error for the eigenvectors. The reported bound, however, has two main limitations: it relies on simplicity of the spectrum, which is a strict assumption, and it requires as input an over-estimate on the eigenvector condition number of the problem, which is unknown, and [15] does not describe how to compute it (see Appendix B.1 for more details). In this work we describe how to overcome these limitations and provide a novel, end-to-end, provably accurate analysis (in the sense of Eq. (2)) for arbitrary invariant subspaces of definite GEPs with ˜ O ( T MM ( n )) boolean complexity.

1.3 Contributions and methods

Our main contribution, summarized in the following Theorem 1.1 and Algorithm 1, is an end-to-end analysis to solve Problem 1.1 in nearly O ( T MM ( n )) time.

Theorem 1.1. Let ( H , S ) be a Hermitian definite pencil of size n with ∥ H ∥ , ∥ S -1 ∥ ≤ 1 , λ 1 ≤ λ 2 ≤ . . . ≤ λ n its eigenvalues, gap k = λ k +1 -λ k and κ ( S ) = ∥ S ∥∥ S -1 ∥ . Algorithm 1

takes as inputs H , S , an integer k ∈ [ n -1] , an error parameter ϵ ∈ (0 , 1) and returns a matrix ˜ Π k such that

where Π k is the true spectral projector on the invariant subspace that is associated with the k smallest (or largest) eigenvalues. The algorithm executes

floating point operations with O ( log( n ) ( log 4 ( n gap k ) + log 4 ( nκ ( S )) + log 3 ( 1 ϵ gap k ) log( κ ( S ) ϵ gap k ) )) bits of precision. Internally, the algorithm needs to generate a total of at most ˜ O ( n ) standard normal floating point numbers using additional O (log(log( n ))) bits.

To achieve the results of Theorem 1.1, we provide a novel O ( T MM ( n )) -type complexity analysis of several problems in numerical linear algebra that can be of independent interest.

In brief, our methodology is as follows. We first observe that if we can determine reasonable 'guesses' for the spectral gap ( ˜ gap k ) and for the midpoint ( ˜ µ k ) between the λ k and λ k +1 eigenvalues then we can efficiently compute the spectral projector by approximating the sign function

using the analysis of [15] for the Newton iteration. The matrix 1 2 ( I + sgn( ˜ µ k -S -1 H )) indeed transforms in exact arithmetic all eigenvalues that are smaller than ˜ µ k to 1 and the ones larger than ˜ µ k to zero. As will be proved in Proposition 2.1, in Section 2, this approach is sufficient to provide an accurate spectral projector ˜ Π k in floating point. As a consequence, the problem reduces to approximating the aforementioned midpoint and gap. As a baseline, in Appendix B.1 we prove that this can be done in nearly O ( T MM ( n )) with iterative inversion [39] and diagonalization [15] or, similarly, by iteratively calling generalized diagonalization [41]. However, this approach presents two drawbacks: It does not take advantage of the inherent symmetry of the problem, and, at the same time, it performs a full diagonalization when we are only interested in the gap between two

specific eigenvalues, which is seemingly redundant. We formally prove this claim by designing a novel approach that achieves better complexity, typically by a factor of O (log( n )) (cf. Section 3.4). Importantly, no explicit diagonalization is necessary.

To minimize the complexity of our algorithm, it is crucial to leverage symmetry. To that end we use the Cholesky factorization of S in the spirit of the Cholesky-QR algorithm [37]. We highlight that, while other factorizations have been solved in O ( T MM ( n )) in floating point, an end-to-end analysis for Cholesky remains open. In exact arithmetic, for example, the LU of a Hermitian definite matrix directly provides its Cholesky and [39] showed that the LU factorization of non-symmetric matrices can be obtained in O ( T MM ( n )) . However, when considering arithmetic errors, the relationship between LU and Cholesky does not hold in floating point, as demonstrated by the counter-example of Appendix C.5. Other fast Cholesky algorithms have been proposed for special classes of matrices, e.g., for matrices with well-defined separators [58, 94, 61] and graph Laplacians [91, 92]. However, they do not generalize to arbitrary dense matrices. Our analysis is the first to improve the classic O ( n 3 ) floating point Cholesky algorithms [84, 71] for the general case, with provable error bounds. In the following Theorem 1.2 we summarize our new analysis of the Cholesky factorization Algorithm 2 (see also Appendix C). We note that the algorithm itself is not new, only its analysis.

Theorem 1.2. Given a Hermitian positive-definite matrix M , there exists an algorithm L ← CHOLESKY ( M ) , listed in Algorithm 2, which requires O ( T MM ( n )) arithmetic operations. This algorithm is logarithmically stable, in a sense that, there exist global constants c 1 , c 2 , c 3 , such that for all ϵ ∈ (0 , 1) , if executed in a floating point machine with precision

which translates into O ( log( n ) log( κ ( M )) + log( 1 ϵ ) ) required bits of precision, then it does not break down due to arithmetic errors, and the solution returned satisfies ∥ LL ∗ -M ∥ ≤ ϵ ∥ M ∥ .

This stand-alone result fulfills the definition of 'logarithmic-stability,' a notion of numerical stability that is commonly used in the related literature [39, 15]. Given this new Cholesky analysis, the following transformation of the GEP to a regular Hermitian eigenvalue problem:

can be carried out accurately in O ( T MM ( n )) in floating point, with provable forward-error bounds for all eigenvalues of the transformed problem. Here, L is the Cholesky factor of S -1 instead of S . Specifically, in Proposition C.3 in Appendix C.4, we prove that the corresponding Algorithm 4, ˜ H ← REDUCE ( H , S , ϵ ) , returns a Hermitian matrix ˜ H such that, for any given accuracy ϵ ∈ (0 , 1) ,

The symmetry induced by the Cholesky transformation is crucial to design an efficient algorithm for the spectral gap. As described in Section 3, and analyzed in Appendices D and E, any spectral gap or eigenvalue of a Hermitian definite pencil can be approximated by an iterative algorithm that uses only 'counting-queries', i.e., queries that ask how many eigenvalues are smaller than a given threshold. This way we completely avoid diagonalization, thus leading to a lower complexity.

To perform the counting queries efficiently, the transformed matrix ˜ H must be regularized with small random perturbations, in the spirit of smoothed analysis [128], which has recently drawn attention in the context of matrix algorithms [31, 25, 15, 41, 104] (see Appendix D for the analysis). These aforementioned works typically require a guarantee on the minimum eigenvalue gap of the perturbed matrix, e.g., [104] uses a Minami-type bound [105], while in [15, 41] the entire pseudospectrum of the perturbed matrix must be shattered with respect to a grid. The latter is even more challenging to achieve than a minimum gap and it requires ˜ O ( n 2 ) random bits. Our algorithm is significantly less demanding in terms of randomness: All we need is the Wegner estimate [138, 4] for the density-ofstates of random Hermitian operators, and only ˜ O ( n ) random bits in total.

Finally, in Section 4, we apply our main results to prove the first matrix multiplication-type upper bounds for the bit complexity of PCA algorithms. Specifically, for the standard PCA formulation, we show that we can first compute the spectral projector and then use deflation to obtain a basis for the desired low-dimensional embedding in nearly matrix multiplication time. We then apply similar arguments to the seminal Block-Krylov PCA algorithm of [106].

PROJECTOR Input: Hermitian definite pencil H ∈ H n , S ∈ H n ++ , gap index k , accuracy ϵ . Requires: ∥ H ∥ ≤ 1 , ∥ S -1 ∥ ≤ 1 , k ∈ [ n -1] . Algorithm: ˜ Π ← PROJECTOR ( H , S , k, ϵ ) . 1: ˜ µ k , ˜ gap k ← GAP ( H , S , k, 1 8 , 1 2 n ) . 2: ˜ κ ← COND ( S , 1 4 , 1 2 n ) . ▷ This is skipped when S = I . 3: ˜ Π ← PURIFY ( H , S , ˜ µ k , ˜ gap k , ˜ κ, ϵ ) . 4: return ˜ Π . Output: Approximate projector ˜ Π on the invariant subspace associated with the k smallest eigenvalues. Ensures: ∥ ˜ Π -Π ∥ ≤ ϵ with probability at least 1 -1 /n .

Algorithm 1: PROJECTOR .

Algorithm 2: CHOLESKY ( M ) .

1.4 Notation

Matrices are denoted by bold capital letters and vectors by bold small letters. For real or complex constants we typically use Greek letters, or the Latin letters c, C . The vector e i denotes the i -th column of the standard basis. A ∗ is the conjugate transpose of A and A † denotes the pseudoinverse. The 2-norm is the default for matrices and vectors. κ ( A ) = ∥ A ∥∥ A † ∥ is the two-norm condition number of A . For the error analysis of the various algorithms, we use E OP i to denote the error matrices that are introduced by the floating point errors of the i -th operation OP . The letters ϵ and δ typically denote (scalar) error quantities and failure probabilities, respectively. [ n ] is the set { 1 , 2 , ..., n } . We denote by H n ⊂ R n the set of Hermitian matrices of size n × n , H n + the set of Hermitian positive semi-definite matrices and H n ++ the set of Hermitian positive definite matrices. For a matrix A and a scalar z we write z ± A as a shorthand for z I ± A . Λ( A ) and Λ( A , B ) denote the spectrum of a matrix A and a matrix pencil ( A , B ) , respectively. The eigenvalues and

singular values are always sorted in ascending order by default: λ 1 ≤ λ 2 ≤ . . . ≤ λ n . Λ ϵ ( A ) is the ϵ -pseudospectrum of A (see Definition A.2).

2 Computing spectral projectors with the sign function

Given a Hermitian definite pencil ( H , S ) , our ultimate goal is to compute a forward error approximation of the spectral projector that is associated with the k smallest eigenvalues, as described in Problem 1.1. Algorithm 3 solves this problem provably and efficiently, but it requires that we already have a suitable approximation of the eigenvalue gap that separates the desired subspace from the rest of the eigenspace. The algorithm is called PURIFY since it is inspired by 'purification' techniques in DFT, referring to the removal of the unoccupied orbitals. The computation of the gap and the midpoint is in fact the bottleneck of our main algorithm, however, we still show that they can be computed efficiently in Section 3, and, importantly, without diagonalizing any matrices. The properties of Algorithm 3 are stated in Proposition 2.1.

Proposition 2.1. Let H ∈ H n with ∥ H ∥ ≤ 1 , S ∈ H n ++ with ∥ S -1 ∥ ≤ 1 , k ∈ [ n -1] and ϵ ∈ (0 , 1) . Let µ k = λ k + λ k +1 2 and gap k = λ k -λ k +1 , where λ 1 ≤ . . . ≤ λ n are the generalized eigenvalues of the Hermitian definite pencil ( H , S ) and assume that we want to compute Π k which is the true spectral projector associated with the k smallest eigenvalues. If we have access to

for some constant C > 1 , then Algorithm 3 computes ˜ Π k ← PURIFY ( H , S , ˜ µ k , ˜ gap k , ˜ κ, ϵ ) such that ∥ ˜ Π k -Π k ∥ ≤ ϵ, in O ( T MM ( n ) ( log( 1 gap k ) + log(log( κ ( S ) ϵ gap k )) )) floating point operations using O ( log( n ) log 3 ( 1 gap k ) log( κ ( S ) ϵ gap k ) ) bits of precision.

Proof. The full proof of Proposition 2.1 can be found in Appendix B. We briefly summarize it here. The main idea is to use the sign function algorithm from [15], SGN , to approximate sgn( ˜ µ k -S -1 H ) . If we already know that ˜ µ k is a reasonable approximation of µ k , and that it is located inside the correct eigenvalue gap, then, in exact arithmetic, our problem is equivalent to computing sgn( µ k -S -1 H ) . The result can be used to filter the desired spectral projector, often referred as 'purification' in the context of DFT. The main challenge is to ensure that all propagated numerical errors, success probabilities, and input parameters for all algorithms are well-defined and bounded. To obtain the final forward errors we must rely on matrix similarity arguments, the properties of the pseudospectrum, the eigenvalue bounds of Weyl and Kahan from Fact A.1, and Lemma B.1, which gives explicit bounds on the sign function under small floating point perturbations.

The rest of the paper is devoted to the analysis of our new algorithm for the spectral gap and the midpoint based on eigenvalue counting queries, described in Theorem 3.1. For comparison purposes, in Appendix B.1 we analyze a diagonalization-based algorithm for the same task (which is a new result itself), specifically, using the state-of-the-art EIG algorithm of [15]. We compare the two algorithms in Section 3.4, demonstrating that our counting-based algorithm is indeed faster.

3 Fast spectral gaps with counting queries

Our core algorithm efficiently approximates spectral gaps based on 'eigenvalue counting queries' only, thus avoiding an explicit (and expensive) diagonalization. To give some intuition on the main idea, consider the following simplified version of the problem.

Problem 3.1 (Gap finder) . Let λ 1 ≤ . . . ≤ λ n in [ -1 , 1] be n (unknown) real values (e.g., they can be the eigenvalues of the original matrix pencil) µ k = λ k + λ k +1 2 , and gap k = λ k +1 -λ k , for some k ∈ [ n -1] . Given k and some error parameter ϵ ∈ (0 , 1 / 2) as input, we want to approximate µ k and gap k up to additive ϵ gap k , i.e., we look for ˜ µ k = µ k ± ϵ gap k and ˜ gap k ∈ (1 ± ϵ ) gap k . Only queries of the following form can be performed: We fix a parameter γ ∈ (0 , 1 / 2) , which distorts all λ i to some (unknown) λ ′ i ∈ [ λ i -γ, λ i + γ ] . We can then choose any value h ∈ [ -1 -γ, 1 + γ ] and ask how many values λ ′ i are smaller than h . For each γ , we can query arbitrarily many different values for h , and each h -query costs q (1 /γ ) = O (polylog(1 /γ )) .

The query cost is arbitrary to avoid trivial solutions by setting γ = 0 . The following proposition is proved in Appendix E:

Proposition 3.1. Problem 3.1 can be solved iteratively by executing a total of O (log( 1 ϵ gap k )) iterations and Θ(1) queries per iteration, where each query costs at most q ( 1 ϵ gap k ) .

3.1 Smoothed analysis of eigenvalue counting

To use the counting query model of Problem 3.1 and Proposition 3.1 to compute the spectral gap of a matrix pencil, we need a 'black-box' method to count eigenvalues that are smaller than a threshold. We first describe a straightforward, deterministic algorithm COUNT ( ˜ X , h, ε ) for this task (see Lemma E.1), which takes as input a Hermitian matrix ˜ X , a scalar h , and a parameter ε , with the requirement that σ min ( h -˜ X ) > ε . It returns the precise number of eigenvalues of ˜ X that are smaller than h . The runtime of the algorithm depends on log(1 /ε ) , and must therefore be minimized. For this we resort to smoothed analysis: We apply a random perturbation to ensure that ε is at least polynomial in 1 /n , up to some other factors detailed in Appendix D.

To build a random 'regularizer,' in Definition D.1 we introduce a random oracle that samples numbers from a standard normal distribution and returns their floating point representation using a prespecified number of bits. Based on this simple oracle, we can design a floating point algorithm ˜ X ← REGULARIZE ( A , γ, δ ) which has the following properties described in Proposition 3.2:

Proposition 3.2. Let A with ∥ A ∥ ≤ 1 be a Hermitian matrix, γ, δ ∈ (0 , 1 / 4) two given parameters, and ˜ X ← REGULARIZE ( A , γ, δ ) . Let g be an arbitrary (but fixed) grid of points in [ -2 , 2] with cardinality | g | = T . For every element h i ∈ g consider the matrices M i = h i -˜ X and ˜ M = h i -˜ X + E i , where E i denote the diagonal floating point error matrices induced by the shift. All the following hold simultaneously with probability 1 -2 δ if we use O (log( Tn γδ )) bits of precision:

Proof. The main result that we use in the proof can be traced back to the Wegner estimate for the density-of-states of Hermitian operators under random diagonal disorder [138]. See Appendix D and in particular D.2 for more details.

3.2 Computing the gap and the midpoint

We can now describe the algorithm GAP in Theorem 3.1, that computes the k -th gap and the midpoint of a Hermitian definite pencil. The same methodology can be extended to approximate any singular value, as described in Proposition E.2 in Appendix E.3.

Theorem 3.1 ( GAP ) . Let H ∈ H n , S ∈ H n ++ and ∥ H ∥ , ∥ S -1 ∥ ≤ 1 , which define a Hermitian definite pencil ( H , S ) . Given k ∈ [ n -1] , accuracy ϵ ∈ (0 , 1) , and failure probability δ ∈ (0 , 1 / 2) , there exists an algorithm

which returns ˜ µ k = µ k ± ϵ gap k and ˜ gap k = (1 ± ϵ ) gap k , where µ k = λ k + λ k +1 2 and gap k = λ k -λ k +1 . The algorithm requires

arithmetic operations using O ( log( n ) ( log 4 ( n δϵ gap k ) + log( κ ( S )) )) bits, where λ i are the eigenvalues of ( H , S ) . If κ ( S ) is unknown, additional O ( T MM ( n ) log( nκ ( S ) δ ) log( κ ( S ))) floating point operations and O (log( n ) log 4 ( nκ ( S ) δ )) bits are sufficient to compute it with Corollary E.1.

Proof. The full proof builds upon the results that are detailed in Appendices D and E. A summary is the following. We first fix our initial error parameter γ 0 = 1 / 8 and call ˜ H = REDUCE ( H , S , γ 0 4 )

(Algorithm 4), which internally uses CHOLESKY to reduce the GEP to a regular Hermitian one. From Proposition C.3, a Hermitian matrix ˜ H is returned such that | λ i ( ˜ H ) -λ i ( H , S ) | ≤ γ 0 4 .

Next, we use the same counting query model as in Proposition 3.1. We first regularize ˜ H using ˜ X ← REGULARIZE ( ˜ H , γ 0 2 , δ 0 ) , where δ 0 = δ/ 2 is the initial failure probability. Conditioning on success of Proposition 3.2 (with probability 1 -δ 0 ) , for all i ∈ [ n ] , it holds that | λ i ( ˜ X ) -λ i ( ˜ H ) | ≤ 9 γ 0 / 16 . Summing the two eigenvalue error bounds, we conclude that all eigenvalues of ( H , S ) , which initially lie in [ -1 , 1] , are distorted by at most γ 0 in ˜ X . We now have all necessary tools to go back to the counting query model of Proposition 3.1: In the first step we construct a grid g = {-1 , -7 / 8 , -6 / 8 , . . . , 7 / 8 , 1 , 9 / 8 } . Clearly, | g | = Θ(1) . Since we conditioned on the success of Proposition 3.2, the regularization ensures that for every h j ∈ g it holds that σ min ( h j -˜ X + E ) ≥ ε 0 with ε 0 = γ 0 δ 0 8 | g | n √ π ln(4 n/δ 0 ) . This allows us to efficiently execute COUNT ( ˜ X , h j , ε 0 ) for every h j .

At the end of the first iteration, we have computed two intervals I k and I k +1 , where I k contains λ k and I k +1 contains λ k +1 , and each interval has size at most 1 , i.e., half the size of [ -1 , 1] . We continue by halving at each step both γ and δ , constructing the corresponding grids as per the proof of Proposition 3.1, and counting eigenvalues over the grid. In each iteration after the first one, we keep track of two intervals I k and I k +1 , and two corresponding grids g k and g k +1 with size | g k | = | g k +1 | = Θ(1) . We therefore only need to execute a constant number of COUNT queries, and in each iteration the size of the intervals I k and I k +1 is halved. The algorithm terminates after a total of m = O ( log( 1 ϵ gap k ) ) iterations and finally provides the advertised complexity, bit requirements, failure probability, and approximation guarantees.

3.3 Sketch proof of Theorem 1.1

The proof of our main Theorem 1.1 directly follows from Theorem 3.1 together with Proposition 2.1 as well as the algorithm SIGMAK (described in Appendix E.3) which is used to compute the condition number of S . The full proof can be found in Appendix E.4.

3.4 Comparison with diagonalization

We can now compare Theorem 3.1 with a diagonalization-based approach that is detailed in Proposition B.2. We fix δ = O (1 /n ) so that both algorithms succeed with the same probability.

For ϵ, gap k , κ -1 ( S ) ∈ Ω(poly(1 /n )) , the total arithmetic complexity of the algorithm of Theorem 3.1 is O ( T MM ( n ) log 2 ( n )) using O (log 5 ( n )) bits. For the same parameters, Proposition B.2 requires need a total of O ( T MM ( n ) log 3 ( n )) arithmetic operations, and O (log 5 ( n )) bits. Thus, in total, Algorithm 3.1 is faster by a factor of O (log( n )) .

In the extreme case where ϵ, gap k , κ ( S ) = Θ(1) , Theorem 3.1 counts O ( T MM ( n ) log( n )) arithmetic operations and O (log 5 ( n )) bits, while Proposition B.2 requires O ( T MM ( n ) log 2 ( n )) operations, and O (log 5 ( n )) bits. Thus, Proposition B.2 is again slower by a factor of O (log( n )) . Interestingly, in this case Theorem 3.1 is faster than even a single call to EIG , which requires O ( T MM ( n ) log 2 ( n )) arithmetic operations. We conclude that, at least based on the currently existing algorithms, diagonalization is redundant for the computation of spectral gaps and invariant subspaces.

3.5 Application in DFT

In Appendix F we demonstrate how our main results can be directly applied to approximate density matrices and electron densities of atomic systems in DFT. Even though is not a machine learning problem per se, we decided to dedicate a section in the Appendix due to its importance: DFT calculations persistently occupy supercomputing clusters and the corresponding software libraries and literature receive tens of thousands of citations annually at the time of this writing [89, 59, 60, 127, 74]. Our work is the first analysis to provide forward-error guarantees in finite precision for these problems in nearly matrix multiplication time.

4 PCA

Since its introduction in the early twentieth century [116, 72], Principal Component Analysis is one of the most important tools in statistics, data science, and machine learning. It can be used, for example, to visualize data, to reduce dimensionality, or to remove noise from data; cf. [79, 45] for reviews on the vast bibliography. In its simplest formulation, given a (centered) data matrix X ∈ R m × n , the goal is to learn a k -dimensional embedding C k , where k < n , that maximizes the sample variance, which can be written as an optimization problem

where H = X ⊤ X ∈ R n × n is the sample covariance. It can be shown that the solution C k corresponds to the principal k singular vectors of H , i.e. the ones that correspond to the largest k singular values. Evidently, since the sample covariance is always symmetric and positive semi-definite, this can be written as a Hermitian eigenvalue problem

(which is indeed a definite GEP as in Equation (1) with S = I ). By solving for C k , we can project the data in k dimensions by computing XC k , preserving as much of the variance in k dimensions as possible. To compute C k we can directly use our main results. However, the solution of Equation (3) is an actual orthonormal basis for the invariant subspace rather than the spectral projector that Theorem 1.1 returns. This can be addressed with deflation: Once we have the spectral projector ˜ Π k , assuming that the approximation is sufficiently tight, we can apply a subsequent deflation step based on a rank-revealing QR factorization to obtain a k -dimensional basis. This can be done deterministically in O ( n 3 ) time [66] or in randomized O ( n ω ) [39, 15].

In Appendix G.1 we prove the following Theorem 4.1 for Algorithm 7, which builds upon our main Theorem 1.1, the algorithm of Proposition E.2 to approximate ∥ X -X k ∥ , and the DEFLATE algorithm of [19], to solve the standard PCA problem of Eq. (3). Following the existing literature, the result is stated for real matrices, but it can be trivially adapted to the complex case as well.

Theorem 4.1 (PCA) . Let X ∈ R m × n be a centered data matrix, H the n × n symmetric sample covariance matrix, i.e., H = X ⊤ X , ∥ H ∥ ≤ 1 , k ∈ [ n ] a target rank, and ϵ ∈ (0 , 1) an accuracy parameter. Given H , we can compute a matrix ˜ C k with k columns such that ∥ X -X ˜ C k ˜ C ⊤ k ∥ ≤ (1 + ϵ ) ∥ X -XC k C ⊤ k ∥ , where C k ∈ R n × k contains the topk (right) singular vectors of X in

arithmetic operations using O ( log( n ) ( log 4 ( n ϵ gap k ) + log 4 ( n σ k +1 ) ) +log( 1 ϵσ k +1 ) ) bits of precision, with probability at least 1 -O (1 /n ) .

4.1 Block-Krylov PCA

In some applications, the target dimension k might be small, i.e., k ≪ n . This condition has driven a whole area of research in so-called low-rank approximation algorithms for PCA [53, 120, 33, 67, 100, 106, 30, 29, 5]. Such approaches are also suitable for kernel PCA, since they rely on matrix-vector products and therefore the kernel matrix does not need to be explicitly formed. The techniques from the previous section can be directly applied to obtain new bit complexity upper bounds for existing state-of-the-art algorithms, which are typically analyzed in exact arithmetic. They internally rely on the computation of the principal singular vectors of submatrices, which can be improved with our methods. Specifically, in Appendix G.2 we summarize a floating point analysis of the Block-Krylov Iteration algorithm (see Algorithm 8), essentially, providing a matrix multiplication-type upper bound on the bit complexity with only a polylogarithmic dependence on the singular value gap. In a nutshell, we directly obtain the following result:

Theorem 4.2 (Bit complexity analysis of Block-Krylov PCA) . Let X be a data matrix X ∈ R m × n , ∥ X ∥ ≤ 1 , k ∈ [ n ] a target rank, ϵ PCA ∈ (0 , 1) an accuracy parameter, and q = Θ ( log( n ) √ ϵ PCA ) . Let T MMX ( k ) denote the complexity to stably multiply X or X ⊤ with a dense matrix with k columns

from the right (see Def. G.1). Using the Steps 1-6 that are detailed in Appendix G.3 as a floating point implementation of Algorithm 8, we can compute a matrix ˜ Z k ∈ R m × k that satisfies

with high probability, where Z k is an approximate basis for the topk principal components of X , returned by Algorithm 8 in exact arithmetic. The total cost is at most

floating point operations, using O ( polylog( mqκ ( K ) ϵ PCA gap k ) ) bits of precision. K , M are as in Alg. 8.

Proof. The full proof can be found in Thm. G.2, Appendix G.3. The main idea is to apply the counting query methodology to compute the condition number of the Block-Krylov matrix K , as well as the k -th spectral gap and the midpoint of the reduced matrix M in Line 5 of Alg. 8. Thereafter, we can compute a spectral projector and an approximate basis for the topk singular vectors of M using PURIFY and DEFLATE , similar to the analysis of classical PCA in the previous section.

5 Open problems

We mention some open problems and interesting future directions.

(i) Bit requirement of SGN : The major bottleneck for the bit requirements of our main algorithms comes from the SGN algorithm of [15]. An inverse-free Newton-Schultz iteration [83], or the implicit repeated squaring of [41, 121] can potentially give significant improvements.

(ii) Sparse algorithms: In applications like DFT it commonly occurs that the matrices have special structure, i.e., they are banded and/or sparse. It remains open whether Problem 1.1 can be provably solved faster than our reported results in finite precision for these special cases (recall that the tridiagonal QR algorithm requires O ( n 3 ) operations to return the eigenvectors). An end-to-end stability analysis of existing fast eigensolvers would be the place to start [65, 137].

(iii) Distributed PCA: The techniques for Block-Krylov PCA can be potentially applied to distributed or streaming PCA algorithms, which are also based on randomized low-rank approximations. E.g., in the distributed PCA algorithm of [31], it is straightforward to replace the SVD computation on the server by a counting query iteration. The full analysis of such an approach is left as future work.

6 Conclusion

In this work we provided an end-to-end analysis to approximate spectral projectors on k -dimensional invariant subspaces of Hermitian definite matrix pencils ( H , S ) that require at most O ( T MM ( n ) polylog( n, ϵ -1 , κ ( S ) , gap -1 k ) ) bit operations in the floating point model of computation. This is the first end-to-end analysis that improves the ˜ O ( n 3 ) complexity of classic eigensolvers for both the regular and the generalized case. To achieve this result we introduced a new method to approximate spectral gaps by querying the number of eigenvalues that are smaller than a threshold, and therefore completely avoid an explicit diagonalization of any matrix or pencil. This approach required proving that the Cholesky factorization can be stably computed in O ( T MM ( n )) floating point operations, a novel result per se . Our results have direct implications on PCA problems, providing matrix multiplication type upper bounds for the bit complexity of classical and Block-Krylov PCA.

References

Dimitris Achlioptas. Database-friendly random projections. In Proc. Twentieth ACM SIGMOD-SIGACT-SIGART Symposium on Principles of Database Systems , pages 274-281, 2001. 61

Satoru Adachi, Satoru Iwata, Yuji Nakatsukasa, and Akiko Takeda. Solving the trust-region subproblem by a generalized eigenvalue problem. SIAM Journal on Optimization , 27(1):269291, 2017. 26

Nir Ailon and Bernard Chazelle. Approximate nearest neighbors and the fast JohnsonLindenstrauss transform. In Proc. Thirty-Eighth Annual ACM Symposium on Theory of Computing , pages 557-563, 2006. 62

Michael Aizenman, Ron Peled, Jeffrey Schenker, Mira Shamis, and Sasha Sodin. Matrix regularizing effects of Gaussian perturbations. Communications in Contemporary Mathematics , 19(03):1750028, 2017. 4, 47

Zeyuan Allen-Zhu and Yuanzhi Li. LazySVD: Even faster SVD decomposition yet without agonizing pain. Advances in Neural Information Processing Systems , 29, 2016. 9

Josh Alman, Ran Duan, Virginia Vassilevska Williams, Yinzhan Xu, Zixuan Xu, and Renfei Zhou. More asymmetry yields faster matrix multiplication. arXiv preprint arXiv:2404.16349 , 2024. 2

Diego Armentano, Carlos Beltr´ an, Peter B¨ urgisser, Felipe Cucker, and Michael Shub. A stable, polynomial-time algorithm for the eigenpair problem. Journal of the European Mathematical Society , 20(6):1375-1437, 2018. 26

Zhaojun Bai and James Demmel. On a direct algorithm for computing invariant subspaces with specified eigenvalues . University of Tennessee, Computer Science Department, 1991. 2, 26

Zhaojun Bai and James Demmel. Using the matrix sign function to compute invariant subspaces. SIAM Journal on Matrix Analysis and Applications , 19(1):205-225, 1998. 2, 26, 27

Zhaojun Bai, James Demmel, and Ming Gu. An inverse free parallel spectral divide and conquer algorithm for nonsymmetric eigenproblems. Numerische Mathematik , 76(3):279308, 1997. 2, 26

G¨ okhan H Bakır, Jason Weston, and Bernhard Sch¨ olkopf. Learning to find pre-images. Advances in Neural Information Processing Systems , 16:449-456, 2004. 1

Pierre Baldi and Kurt Hornik. Neural networks and principal component analysis: Learning from examples without local minima. Neural networks , 2(1):53-58, 1989. 1

Grey Ballard, James Demmel, and Ioana Dumitriu. Minimizing Communication for Eigenproblems and the Singular Value Decomposition. Technical Report UCB/EECS-2011-14 , February 2011. 3

Grey Ballard, James Demmel, Olga Holtz, and Oded Schwartz. Minimizing Communication in Numerical Linear Algebra. SIAM Journal on Matrix Analysis and Applications , 32(3):866901, 2011. 3

Jess Banks, Jorge Garza-Vargas, Archit Kulkarni, and Nikhil Srivastava. Pseudospectral Shattering, the Sign Function, and Diagonalization in Nearly Matrix Multiplication Time. Foundations of Computational Mathematics , pages 1-89, 2022. 2, 3, 4, 6, 9, 10, 19, 21, 22, 23, 26, 27, 31, 32, 33

Jess Banks, Jorge Garza-Vargas, and Nikhil Srivastava. Global Convergence of Hessenberg Shifted QR II: Numerical Stability. arXiv preprint arXiv:2205.06810 , 2022. 26

Jess Banks, Jorge Garza-Vargas, and Nikhil Srivastava. Global Convergence of Hessenberg Shifted QR III: Approximate Ritz Values via Shifted Inverse Iteration. arXiv preprint arXiv:2205.06804 , 2022. 26

Jess Banks, Jorge Garza-Vargas, and Nikhil Srivastava. Global Convergence of Hessenberg Shifted QR I: Exact Arithmetic. Foundations of Computational Mathematics , pages 1-34, 2024. 26

Jess Banks, Archit Kulkarni, Satyaki Mukherjee, and Nikhil Srivastava. Gaussian Regularization of the Pseudospectrum and Davies' Conjecture. Communications on Pure and Applied Mathematics , 74(10):2114-2131, 2021. 9, 62

Elnaz Barshan, Ali Ghodsi, Zohreh Azimifar, and Mansoor Zolghadri Jahromi. Supervised principal component analysis: Visualization, classification and regression on subspaces and submanifolds. Pattern Recognition , 44(7):1357-1371, 2011. 1

Yair Bartal, Ben Recht, and Leonard J Schulman. Dimensionality reduction: beyond the Johnson-Lindenstrauss bound. In Proc. 2011 Annual ACM-SIAM Symposium on Discrete Algorithms , pages 868-887. SIAM, 2011. 62

Harold Basch, C. J. Hornback, and J. W. Moskowitz. Gaussian-Orbital Basis Sets for the FirstRow Transition-Metal Atoms. The Journal of Chemical Physics , 51(4):1311-1318, 1969. 58

Robert A Baston and Yuji Nakatsukasa. Stochastic diagonal estimation: probabilistic bounds and an improved algorithm. arXiv preprint arXiv:2201.10684 , 2022. 62

Friedrich L Bauer and Charles T Fike. Norms and exclusion theorems. Numerische Mathematik , 2:137-141, 1960. 20

Michael Ben-Or and Lior Eldar. A Quasi-Random Approach to Matrix Spectral Analysis. In Proc. 9th Innovations in Theoretical Computer Science Conference , pages 6:1-6:22. Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik, 2018. 2, 4, 26

Peter Benner, Volker Mehrmann, and Hongguo Xu. A new method for computing the stable invariant subspace of a real Hamiltonian matrix. Journal of computational and applied mathematics , 86(1):17-43, 1997. 26

Rajendra Bhatia. Pinching, trimming, truncating, and averaging of matrices. The American Mathematical Monthly , 107(7):602-608, 2000. 25

Rajendra Bhatia. Perturbation bounds for matrix eigenvalues . SIAM, 2007. 20

Srinadh Bhojanapalli, Prateek Jain, and Sujay Sanghavi. Tighter low-rank approximation via sampling the leveraged element. In Proc. Twenty-Sixth Annual ACM-SIAM Symposium on Discrete Algorithms , pages 902-920. SIAM, 2014. 9

Christos Boutsidis, Petros Drineas, and Malik Magdon-Ismail. Near-optimal column-based matrix reconstruction. SIAM Journal on Computing , 43(2):687-717, 2014. 9

Christos Boutsidis, David P Woodruff, and Peilin Zhong. Optimal principal component analysis in distributed and streaming models. In Proc. Forty-Eighth Annual ACM Symposium on Theory of Computing , pages 236-249, 2016. 4, 10

Tim Clark and Rainer Koch. Linear Combination of Atomic Orbitals , pages 5-22. Springer Berlin Heidelberg, 1999. 58

Kenneth L Clarkson and David P Woodruff. Numerical linear algebra in the streaming model. In Proc. Forty-First Annual ACM Symposium on Theory of Computing , pages 205-214, 2009. 9, 64

Kenneth L Clarkson and David P Woodruff. Low-rank approximation and regression in input sparsity time. Journal of the ACM , 63(6):1-45, 2017. 64

Michael B Cohen, Sam Elder, Cameron Musco, Christopher Musco, and Madalina Persu. Dimensionality reduction for k-means clustering and low rank approximation. In Proc. FortySeventh Annual ACM Symposium on Theory of Computing , pages 163-172, 2015. 1

Charles R. Crawford. Reduction of a band-symmetric generalized eigenvalue problem. Communications of the ACM , 16(1):41-44, 1973. 26

Philip I Davies, Nicholas J Higham, and Franc ¸oise Tisseur. Analysis of the cholesky method with iterative refinement for solving the symmetric definite generalized eigenproblem. SIAM Journal on Matrix Analysis and Applications , 23(2):472-493, 2001. 2, 4, 26

Theodorus J Dekker and Joseph F Traub. The shifted QR algorithm for Hermitian matrices. Linear Algebra and its Applications , 4(3):137-154, 1971. 26

James Demmel, Ioana Dumitriu, and Olga Holtz. Fast linear algebra is stable. Numerische Mathematik , 108(1):59-91, 2007. 2, 3, 4, 9, 21, 22, 23, 34, 47, 64

James Demmel, Ioana Dumitriu, Olga Holtz, and Robert Kleinberg. Fast matrix multiplication is stable. Numerische Mathematik , 106(2):199-224, 2007. 2, 21, 22

James Demmel, Ioana Dumitriu, and Ryan Schneider. Generalized Pseudospectral Shattering and Inverse-Free Matrix Pencil Diagonalization. arXiv preprint arXiv:2306.03700 , 2023. 2, 3, 4, 10, 27, 33

James W Demmel. Applied numerical linear algebra . SIAM, 1997. 2, 26

James Weldon Demmel. Three methods for refining estimates of invariant subspaces. Computing , 38(1):43-57, 1987. 26

James Weldon Demmel and Bo K˚ agstr¨ om. Computing stable eigendecompositions of matrix pencils. Linear Algebra and its Applications , 88:139-186, 1987. 26

Konstantinos I Diamantaras and Sun Yuan Kung. Principal component neural networks: theory and applications . John Wiley & Sons, Inc., 1996. 1, 9, 62

Petros Drineas, Iordanis Kerenidis, and Prabhakar Raghavan. Competitive recommendation systems. In Proc. Thiry-Fourth Annual ACM Symposium on Theory of Computing , pages 82-90, 2002. 1

Ran Duan, Hongxun Wu, and Renfei Zhou. Faster matrix multiplication via asymmetric hashing. In 2023 IEEE 64th Annual Symposium on Foundations of Computer Science , pages 2129-2138. IEEE, 2023. 2, 21

Stanley C Eisenstat and Ilse CF Ipsen. Relative perturbation results for eigenvalues and eigenvectors of diagonalisable matrices. BIT Numerical Mathematics , 38(3):502-509, 1998. 20

Ethan N Epperly, Joel A Tropp, and Robert J Webber. XTrace: Making the Most of Every Sample in Stochastic Trace Estimation. SIAM Journal on Matrix Analysis and Applications , 45(1):1-23, 2024. 62

Dan Feldman, Melanie Schmidt, and Christian Sohler. Turning big data into tiny data: Constant-size coresets for k-means, PCA, and projective clustering. SIAM Journal on Computing , 49(3):601-657, 2020. 1

John GF Francis. The QR transformation a unitary analogue to the LR transformation-Part 1. The Computer Journal , 4(3):265-271, 1961. 26

John GF Francis. The QR transformation-Part 2. The Computer Journal , 4(4):332-345, 1962. 26

Alan Frieze, Ravi Kannan, and Santosh Vempala. Fast monte-carlo algorithms for finding low-rank approximations. In Proc. 39th Annual Symposium on Foundations of Computer Science , page 370, 1998. 9, 64

Alan Frieze, Ravi Kannan, and Santosh Vempala. Fast Monte-Carlo algorithms for finding low-rank approximations. Journal of the ACM , 51(6):1025-1041, 2004. 64

Martin F¨ urer. Faster integer multiplication. In Proc. Thirty-Ninth Annual ACM Symposium on Theory of Computing , pages 57-66, 2007. 2

Giulia Galli. Linear scaling methods for electronic structure calculations and quantum molecular dynamics simulations. Current Opinion in Solid State and Materials Science , 1(6):864874, 1996. 59

Giulia Galli and Michele Parrinello. Large scale electronic structure calculations. Physical Review Letters , 69(24):3547, 1992. 59

Alan George. Nested dissection of a regular finite element mesh. SIAM Journal on Numerical Analysis , 10(2):345-363, 1973. 4

P Giannozzi, O Andreussi, T Brumme, O Bunau, M Buongiorno Nardelli, M Calandra, R Car, C Cavazzoni, D Ceresoli, M Cococcioni, N Colonna, I Carnimeo, A Dal Corso, S de Gironcoli, P Delugas, R A DiStasio, A Ferretti, A Floris, G Fratesi, G Fugallo, R Gebauer, U Gerstmann, F Giustino, T Gorni, J Jia, M Kawamura, H-Y Ko, A Kokalj, E K¨ uc ¸ ¨ ukbenli, M Lazzeri, MMarsili, N Marzari, F Mauri, N L Nguyen, H-V Nguyen, A Otero de-la Roza, L Paulatto, S Ponc´ e, D Rocca, R Sabatini, B Santra, M Schlipf, A P Seitsonen, A Smogunov, I Timrov, T Thonhauser, P Umari, N Vast, X Wu, and S Baroni. Advanced capabilities for

materials modelling with Quantum ESPRESSO. Journal of Physics: Condensed Matter , 29(46):465901, oct 2017. 8, 58

Paolo Giannozzi, Stefano Baroni, Nicola Bonini, Matteo Calandra, Roberto Car, Carlo Cavazzoni, Davide Ceresoli, Guido L Chiarotti, Matteo Cococcioni, Ismaila Dabo, Andrea Dal Corso, Stefano de Gironcoli, Stefano Fabris, Guido Fratesi, Ralph Gebauer, Uwe Gerstmann, Christos Gougoussis, Anton Kokalj, Michele Lazzeri, Layla Martin-Samos, Nicola Marzari, Francesco Mauri, Riccardo Mazzarello, Stefano Paolini, Alfredo Pasquarello, Lorenzo Paulatto, Carlo Sbraccia, Sandro Scandolo, Gabriele Sclauzero, Ari P Seitsonen, Alexander Smogunov, Paolo Umari, and Renata M Wentzcovitch. QUANTUM ESPRESSO: a modular and open-source software project for quantum simulations of materials. Journal of Physics: Condensed Matter , 21(39):395502, Sep 2009. 8

John R Gilbert and Robert Endre Tarjan. The analysis of a nested dissection algorithm. Numerische Mathematik , 50(4):377-404, 1986. 4

Stefan Goedecker. Linear scaling electronic structure methods. Reviews of Modern Physics , 71(4):1085, 1999. 59

Stefan Goedecker and GE Scuserza. Linear scaling electronic structure methods in chemistry and physics. Computing in Science & Engineering , 5(4):14-21, 2003. 59

Gene H Golub and Charles F Van Loan. Matrix Computations . Johns Hopkins University Press, 2013. 2, 19, 26, 64

Ming Gu and Stanley C Eisenstat. A divide-and-conquer algorithm for the symmetric tridiagonal eigenproblem. SIAM Journal on Matrix Analysis and Applications , 16(1):172-191, 1995. 10, 59

Ming Gu and Stanley C Eisenstat. Efficient algorithms for computing a strong rank-revealing QR factorization. SIAM Journal on Scientific Computing , 17(4):848-869, 1996. 9

Nathan Halko, Per-Gunnar Martinsson, and Joel A Tropp. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM Review , 53(2):217-288, 2011. 9, 64

David Harvey and Joris Van Der Hoeven. Integer multiplication in time o(nlog \ ,n). Annals of Mathematics , 193(2):563-617, 2021. 2

Qiu He, Bin Yu, Zhaohuai Li, and Yan Zhao. Density Functional Theory for Battery Materials. Energy & Environmental Materials , 2(4):264-279, 2019. 58

Nicholas J Higham. Accuracy and stability of numerical algorithms . SIAM, 2002. 19, 20, 61, 64

Nicholas J Higham. Cholesky factorization. Wiley Interdisciplinary Reviews: Computational Statistics , 1(2):251-254, 2009. 4

Harold Hotelling. Analysis of a complex of statistical variables into principal components. Journal of educational psychology , 24(6):417, 1933. 9, 62

Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations , 2022. 1

J¨ urg Hutter, Marcella Iannuzzi, Florian Schiffmann, and Joost VandeVondele. cp2k: atomistic simulations of condensed matter systems. Wiley Interdisciplinary Reviews: Computational Molecular Science , 4(1):15-25, 2014. 8, 58

Ilse CF Ipsen. Computing an eigenvector with inverse iteration. SIAM Review , 39(2):254291, 1997. 2, 26

Ilse CF Ipsen. Absolute and relative perturbation bounds for invariant subspaces of matrices. Linear Algebra and its Applications , 309(1-3):45-56, 2000. 26

Zhongxiao Jia and GW Stewart. An analysis of the rayleigh-ritz method for approximating eigenspaces. Mathematics of computation , 70(234):637-647, 2001. 26

William B Johnson and Joram Lindenstrauss. Extensions of Lipschitz mappings into a Hilbert space. Contemp. Math. , 26(1):189-206, 1984. 61

Ian T Jolliffe. Principal component analysis for special types of data . Springer, 2002. 1, 9, 62

Praneeth Kacham and David P Woodruff. Faster Algorithms for Schatten-p Low Rank Approximation. Approximation, Randomization, and Combinatorial Optimization. Algorithms and Techniques , 2024. 64

William Kahan. Spectra of nearly Hermitian matrices. Proc. American Mathematical Society , 48(1):11-17, 1975. 20

Daniel M Kane and Jelani Nelson. Sparser Johnson-Lindenstrauss transforms. Journal of the ACM , 61(1):1-23, 2014. 62

Charles S Kenney and Alan J Laub. The matrix sign function. IEEE transactions on automatic control , 40(8):1330-1348, 1995. 10

Andrzej Kiełbasi´ nski. A note on rounding-error analysis of Cholesky factorization. Linear Algebra and its Applications , 88:487-494, 1987. 4

Cedric Klinkert, ´ Aron Szab´ o, Christian Stieger, Davide Campi, Nicola Marzari, and Mathieu Luisier. 2-d Materials for Ultrascaled Field-Effect Transistors: One Hundred Candidates under the Ab Initio Microscope. ACS Nano , 14(7):8605-8615, 2020. 58

Walter Kohn. Density functional/Wannier function theory for systems of very many atoms. Chemical Physics Letters , 208(3-4):167-172, 1993. 59

Walter Kohn. Density functional and density matrix method scaling linearly with the number of atoms. Physical Review Letters , 76(17):3168, 1996. 59

Walter Kohn and Lu Jeu Sham. Self-consistent equations including exchange and correlation effects. Physical Review , 140(4A):A1133, 1965. 2, 58

Georg Kresse and J¨ urgen Furthm¨ uller. Efficient iterative schemes for ab initio total-energy calculations using a plane-wave basis set. Physical Review B , 54(16):11169-11186, 1996. 8, 58

Vera N Kublanovskaya. On some algorithms for the solution of the complete eigenvalue problem. USSR Computational Mathematics and Mathematical Physics , 1(3):637-657, 1962. 26

Rasmus Kyng, Yin Tat Lee, Richard Peng, Sushant Sachdeva, and Daniel A Spielman. Sparsified Cholesky and Multigrid Solvers for Connection Laplacians. In Proc. 48th Annual ACM Symposium on Theory of Computing , pages 842-850, 2016. 4

Rasmus Kyng and Sushant Sachdeva. Approximate Gaussian Elimination for Laplacians Fast, Sparse, and Simple. In Proc. IEEE 57th Annual Symposium on Foundations of Computer Science , pages 573-582, 2016. 4

Kasper Green Larsen and Jelani Nelson. Optimality of the Johnson-Lindenstrauss lemma. In 2017 IEEE 58th Annual Symposium on Foundations of Computer Science , pages 633-638. IEEE, 2017. 62

Richard J Lipton, Donald J Rose, and Robert Endre Tarjan. Generalized nested dissection. SIAM Journal on Numerical Analysis , 16(2):346-358, 1979. 4

Anand Louis and Santosh S Vempala. Accelerated Newton iteration for roots of black box polynomials. In Proc. IEEE 57th Annual Symposium on Foundations of Computer Science , pages 732-740. IEEE, 2016. 2, 3

Arnaz Malhi and Robert X Gao. PCA-based feature selection scheme for machine defect classification. IEEE transactions on instrumentation and measurement , 53(6):1517-1525, 2004. 1

Alexander N Malyshev. Computing invariant subspaces of a regular linear pencil of matrices. Siberian Mathematical Journal , 30(4):559-567, 1989. 26

Alexander N Malyshev. Parallel algorithm for solving some spectral problems of linear algebra. Linear Algebra and its Applications , 188:489-520, 1993. 26

Olvi L Mangasarian and Edward W Wild. Multisurface proximal support vector machine classification via generalized eigenvalues. IEEE transactions on pattern analysis and machine intelligence , 28(1):69-74, 2005. 1

Per-Gunnar Martinsson, Vladimir Rokhlin, and Mark Tygert. A randomized algorithm for the decomposition of matrices. Applied and Computational Harmonic Analysis , 30(1):4768, 2011. 9, 64

Nicola Marzari, Arash A. Mostofi, Jonathan R. Yates, Ivo Souza, and David Vanderbilt. Maximally localized Wannier functions: Theory and applications. Reviews of Modern Physics , 84(4):1419, 2012. 58

Reinhard J Maurer, Christoph Freysoldt, Anthony M Reilly, Jan Gerit Brandenburg, Oliver T Hofmann, Torbj¨ orn Bj¨ orkman, S´ ebastien Leb` egue, and Alexandre Tkatchenko. Advances in Density-Functional Calculations for Materials Modeling. Annual Review of Materials Research , 49(1):1-30, 2019. 58

Istv´ an Mayer. Simple theorems, proofs, and derivations in quantum chemistry . Springer Science & Business Media, 2003. 59

Raphael Meyer, Cameron Musco, and Christopher Musco. On the unreasonable effectiveness of single vector Krylov methods for low-rank approximation. In Proc. 2024 Annual ACMSIAM Symposium on Discrete Algorithms , pages 811-845. SIAM, 2024. 4

Nariyuki Minami. Local fluctuation of the spectrum of a multidimensional Anderson tight binding model. Communications in mathematical physics , 177:709-725, 1996. 4

Cameron Musco and Christopher Musco. Randomized block Krylov methods for stronger and faster approximate singular value decomposition. Advances in Neural Information Processing Systems , 28, 2015. 4, 9, 64, 65

Cameron Musco, Christopher Musco, and Aaron Sidford. Stability of the lanczos method for matrix function approximation. In Proc. Twenty-Ninth Annual ACM-SIAM Symposium on Discrete Algorithms , pages 1605-1624. SIAM, 2018. 2, 3, 24, 64

Yuji Nakatsukasa. On the condition numbers of a multiple eigenvalue of a generalized eigenvalue problem. Numerische Mathematik , 121:531-544, 2012. 26

Yuji Nakatsukasa and Nicholas J Higham. Stable and efficient spectral divide and conquer algorithms for the symmetric eigenvalue decomposition and the SVD. SIAM Journal on Scientific Computing , 35(3):A1325-A1349, 2013. 2

Jelani Nelson and Huy L Nguyˆ en. OSNAP: Faster numerical linear algebra algorithms via sparser subspace embeddings. In Proc. IEEE 54th Annual Symposium on Foundations of Computer Science , pages 117-126. IEEE, 2013. 62

Andrew Ng, Michael Jordan, and Yair Weiss. On spectral clustering: Analysis and an algorithm. Advances in Neural Information Processing Systems , 14, 2001. 1

Christopher C Paige. Error analysis of the lanczos algorithm for tridiagonalizing a symmetric matrix. IMA Journal of Applied Mathematics , 18(3):341-349, 1976. 26

Christopher Conway Paige. The computation of eigenvalues and eigenvectors of very large sparse matrices. PhD thesis, University of London, 1971. 2, 26

Victor Y Pan and Zhao Q Chen. The complexity of the matrix eigenproblem. In Proc. 31st Annual ACM Symposium on Theory of Computing , pages 507-516, 1999. 26

Beresford N Parlett. The Symmetric Eigenvalue Problem . SIAM, 1998. 26

Karl Pearson. Liii. on lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin philosophical magazine and journal of science , 2(11):559-572, 1901. 9, 62

Andrzej Pokrzywa. Spectra of operators with fixed imaginary parts. Proc. American Mathematical Society , 81(3):359-364, 1981. 20

Awwal Mohammed Rufai, Gholamreza Anbarjafari, and Hasan Demirel. Lossy image compression using singular value decomposition and wavelet difference reduction. Digital signal processing , 24:117-123, 2014. 1

Yousef Saad. Numerical methods for large eigenvalue problems: revised edition . SIAM, 2011. 26

Tam´ as Sarl´ os. Improved approximation algorithms for large matrices via random projections. In Proc. 47th Annual Symposium on Foundations of Computer Science , pages 143-152. IEEE, 2006. 9, 64

Ryan Schneider. When is fast, implicit squaring of A -1 B stable? arXiv preprint arXiv:2310.00193 , 2023. 2, 3, 10

Bernhard Sch¨ olkopf, Alexander Smola, and Klaus-Robert M¨ uller. Nonlinear component analysis as a kernel eigenvalue problem. Neural computation , 10(5):1299-1319, 1998. 1

Arnold Sch¨ onhage and Volker Strassen. Fast multiplication of large numbers. Computing , 7:281-292, 1971. 2

Jianbo Shi and Jitendra Malik. Normalized cuts and image segmentation. IEEE Transactions on pattern analysis and machine intelligence , 22(8):888-905, 2000. 1

Max Simchowitz, Ahmed El Alaoui, and Benjamin Recht. Tight query complexity lower bounds for PCA via finite sample deformed wigner law. In Proc. 50th Annual ACM Symposium on Theory of Computing , pages 1249-1259, 2018. 64

Aleksandros Sobczyk and Mathieu Luisier. Approximate Euclidean lengths and distances beyond Johnson-Lindenstrauss. Advances in Neural Information Processing Systems , 35:19357-19369, 2022. 62

Jos´ e M Soler, Emilio Artacho, Julian D Gale, Alberto Garc´ ıa, Javier Junquera, Pablo Ordej´ on, and Daniel S´ anchez-Portal. The SIESTA method for ab initio order-N materials simulation. Journal of Physics: Condensed Matter , 14(11):2745, 2002. 8, 58

Daniel A Spielman and Shang-Hua Teng. Smoothed analysis of algorithms: Why the simplex algorithm usually takes polynomial time. Journal of the ACM , 51(3):385-463, 2004. 4

Nikhil Srivastava. The complexity of diagonalization. In Proc. 2023 International Symposium on Symbolic and Algebraic Computation , pages 1-6, 2023. 26

Gilbert W Stewart. On the sensitivity of the eigenvalue problem ax= λ bx. SIAM Journal on Numerical Analysis , 9(4):669-686, 1972. 26

Gilbert W Stewart. Error and perturbation bounds for subspaces associated with certain eigenvalue problems. SIAM Review , 15(4):727-764, 1973. 26

Gilbert W Stewart. On the perturbation of pseudo-inverses, projections and linear least squares problems. SIAM Review , 19(4):634-662, 1977. 26

Gilbert W Stewart. Pertubation bounds for the definite generalized eigenvalue problem. Linear Algebra and its Applications , 23:69-85, 1979. 26

Volker Strassen. Gaussian elimination is not optimal. Numerische Mathematik , 13(4):354356, 1969. 21

Lloyd N Trefethen and Mark Embree. Spectra and Pseudospectra: The Behavior of Nonnormal Matrices and Operators . Princeton University Press, 2005. 20, 21, 26

Joost VandeVondele, Urban Borstnik, and Jurg Hutter. Linear scaling self-consistent field calculations with millions of atoms in the condensed phase. Journal of Chemical Theory and Computation , 8(10):3565-3573, 2012. 59

James Vogel, Jianlin Xia, Stephen Cauley, and Venkataramanan Balakrishnan. Superfast divide-and-conquer method and perturbation analysis for structured eigenvalue solutions. SIAM Journal on Scientific Computing , 38(3):A1358-A1382, 2016. 10, 59

Franz Wegner. Bounds on the density of states in disordered systems. Zeitschrift f¨ ur Physik B Condensed Matter , 44(1):9-15, 1981. 4, 7, 47

Hermann Weyl. Das asymptotische Verteilungsgesetz der Eigenwerte linearer partieller Differentialgleichungen (mit einer Anwendung auf die Theorie der Hohlraumstrahlung). Mathematische Annalen , 71(4):441-479, 1912. 20

James Hardy Wilkinson. Global convergene of tridiagonal QR algorithm with origin shifts. Linear Algebra and its Applications , 1(3):409-420, 1968. 26

Virginia Vassilevska Williams, Yinzhan Xu, Zixuan Xu, and Renfei Zhou. New bounds for matrix multiplication: from alpha to omega. In Proc. 2024 Annual ACM-SIAM Symposium on Discrete Algorithms , pages 3792-3835. SIAM, 2024. 2, 21

Zewen Xiao and Yanfa Yan. Progress in Theoretical Study of Metal Halide Perovskite Solar Cell Materials. Advanced Energy Materials , 7(22):1701136, 2017. 58

Weitao Yang. Direct calculation of electron density in density-functional theory. Physical Review Letters , 66(11):1438, 1991. 59

Yunkai Zhou, Yousef Saad, Murilo L Tiago, and James R Chelikowsky. Parallel selfconsistent-field calculations via Chebyshev-filtered subspace acceleration. Physical Review E , 74(6):066704, 2006. 59

Yunkai Zhou, Yousef Saad, Murilo L Tiago, and James R Chelikowsky. Self-consistent-field calculations using Chebyshev-filtered subspace iteration. Journal of Computational Physics , 219(1):172-184, 2006. 59

A Preliminaries

In this section the model of computation is defined and basic linear algebra principles are summarized. Further details can be found in standard textbooks such as [70, 64].

A.1 Forward and backward approximation

In numerical analysis, a 'forward-error' often measures the distance between the true solution to a problem and the solution returned by a numerical algorithm. A related notion is the so-called 'backward error' or 'backward approximation.' For more details we can refer to the standard textbook of Higham [70]. In this case, the solution returned is the exact solution of a 'nearby problem,' and the backward error quantifies the distance of the original problem to this nearby problem. In PCA, for example, a backward-approximation could be defined by ˜ C k = arg min C ⊤ C = I k × k tr( C ⊤ ˜ HC ) .

The backward error would be given ∥ H -˜ H ∥ for some norm. An algorithm is backward stable if the backward error is always well-defined and bounded. A forward error type of approximation is often harder to achieve. A common rule-of-thumb states that forward error ≲ backward error × condition number of the problem .

However, this does not generally hold for eigenvalue problems. For example, Proposition 1.1 in [15] can be used to transformed a backward approximate diagonalization error to a forward error for the eigenvectors, but the bound depends on the minimum eigenvalue gap between any eigenvalue pair. As the authors point in Remark 1.2, special treatment is needed in terms of the invariant subspaces in the presence of multiple eigenvalues, which are thoroughly analyzed in this work.

A.2 Floating point model

We assume a standard floating point model of computation and borrow its axioms from [70, Chapter 2]. There is a fixed number of bits to represent floating point numbers, specifically, one bit is reserved for the sign s , p bits are used for the exponent e , and t bits are used for the significand m .

A real number α ∈ R is rounded to a floating point number

The sign s is + if the corresponding bit is one, and -if the bit is zero. The exponent e is stored as a binary number in the so-called biased form , and its range is e ∈ [ -M,M ] , where M = 2 p -1 . The significand m is an integer that satisfies 2 t -1 ≤ m ≤ 2 t -1 , where the lower bound is enforced to ensure that the system is normalized , i.e. the first bit of m is always 1 . We can therefore write fl ( α ) in a more intuitive representation

where the first bit m 1 of m is always equal to one for normalized numbers. The range of normalized numbers is therefore [2 -M , 2 M (2 -2 -t )] . Numbers that are smaller than 2 -M are called subnormal and they will be ignored for simplicity, since we can either add more bits in the exponent, or account for them in the failure probability when the numbers are random (the latter strategy is used for example in Lemma D.1). Similarly, numbers that are larger than 2 M (2 -2 -t ) are assumed to be numerically equal to infinity, denoted by INF .

From [70, Theorem 2.2], for all real numbers α in the normalized range it holds that

where θ ∈ R satisfies | θ | ≤ 2 -t := u , where u is the machine precision . Clearly, t = O (log(1 / u )) , in which case we can always obtain a bound for the number of required bits of a numerical algorithm if we have an upper bound for the precision u . We will write the same for complex numbers which are represented as a pair of normalized floating point numbers.

The floating point implementation of each arithmetic operation ⊙ ∈ { + , -, × , / } also satisfies

Divisions and multiplications with 1 and 2 do not introduce errors (for the latter we simply increase/decrease the exponent). We assume that we also have an implementation of √ · such that fl ( √ α ) = (1 + θ ) √ α where | θ | ≤ u . From [70, Lemma 3.1], we can bound products of errors as

where ρ i = ± 1 and | η n | ≤ n u 1 -n u .

The above can be extended also for complex arithmetic (see [70, Lemma 3.5]), where the bound becomes | θ | ≤ O ( u ) , but we will ignore the constant prefactor for simplicity.

Operations on matrices can be analyzed in a similar manner. Let ⊗ denote the element-wise multiplication between two matrices and ⊘ the element-wise division. The floating point representation of a matrix A satisfies

It can be shown that ∥ ∆ ∥ ≤ u √ n ∥ A ∥ .

For any operation ⊙ ∈ { + , -, ⊗ , ⊘} and matrices A and B it holds that

A.3 Spectral decomposition, pseudospectrum, and eigenvalue bounds

We first recall the definition of the spectral decomposition of a diagonalizable matrix.

Definition A.1 (Spectrum and spectral decomposition) . A matrix A is diagonalizable if there exist invertible matrix V and diagonal matrix Λ such that A = VΛV -1 . This is called the spectral decomposition of A . The set Λ( A ) = { Λ i,i | i = 1 , . . . , n } is the spectrum of A .

The spectral theorem states that Hermitian matrices (or, more generally, normal matrices) can be always diagonalized by unitary transformations.

Theorem A.1 (Spectral theorem) . If A ∈ C n × n is Hermitian, then there exists orthogonal matrix Q ∈ C n × n such that A = QΛQ ∗ , where Λ is a diagonal matrix with real diagonal elements.

In the sections that follow we need to bound the (forward) errors on the eigenvalues of matrices under perturbations that are introduced due to the finite precision arithmetic. Such bounds can be derived by the classic Bauer-Fike theorem the following (cf. [24, 48] for more details).

There are many bounds in the literature describing the effect of perturbations on the eigenvalues. We summarize some classic results in the following proposition and refer to Bhatia's monograph for a detailed overview [28].

Fact A.1. Let A and B be two n × n matrices. The following bounds are known between the eigenvalues λ 1 ( A ) ≤ λ 2 ( A ) ≤ . . . ≤ λ n ( A ) of A and λ 1 ( B ) ≤ . . . ≤ λ n ( B ) of B :

Pseudospectral analysis is useful when the aforementioned bounds are not applicable.

Definition A.2 (Pseudospectrum) . For some ϵ > 0 , the ϵ -pseudospectrum of a matrix M ∈ C n × n is defined as:

where Λ( M ) is the spectrum of M .

Recall some useful properties from the seminal book of Trefethen and Embree [135].

Proposition A.1 (Collective results from [135]) . Let D ( z, r ) denote the open disk of radius r in the complex plane centered at z ∈ C , M and E be two n × n matrices, and ϵ > ∥ E ∥ be a positive real number. The following hold:

(i) Λ ϵ -∥ E ∥ ( M ) ⊆ Λ ϵ ( M + E ) ⊆ Λ ϵ + ∥ E ∥ ( M ) ([135, Thm 52.4]),

(ii) Any bounded connected component of Λ ϵ ( M ) has a nonempty intersection with Λ( M ) , i.e., it contains at least one eigenvalue ([135, Thm 4.3]),

(iii) ⋃ i D ( λ i , ϵ ) ⊆ Λ ϵ ( M ) ⊆ ⋃ i D ( λ i , ϵκ V ( M )) ([135, Thms 4.3 and 52.2]).

A.4 Eigenvector condition number of definite pencils

Some approximation bounds throughout the paper depend on the eigenvector condition number of the generalized eigenproblem (1). For arbitrary GEPs a bound for this quantity might not always exist, but a straightforward bound exists for the Hermitian definite case. To obtain such a bound we define the eigenvector condition number of a diagonalizable matrix as follows:

Proposition A.2. Let H ∈ H n and S ∈ H n ++ , and consider the definite GEP HC = SCΛ . Then

Proof. Since S is Hermitian positive-definite, it can be written as S = LL ∗ for some matrix L . Then we can transform the GEP to a Hermitian eigenproblem, specifically

Since LL ∗ = S we have that ∥ L ∥ 2 = ∥ LL ∗ ∥ = ∥ S ∥ and similarly ∥ L -1 ∥ 2 = ∥ S -1 ∥ , which means that κ ( L ) = √ κ ( S ) . Since L -1 HL -∗ is Hermitian, it can be diagonalized by a unitary matrix, i.e. there exists ̂ C such that ∥ L ∗ ̂ C ∥ = ∥ ( L ∗ ̂ C ) -1 ∥ = 1 and L ∗ ̂ C diagonalizes L -1 HL -∗ . For ̂ C we have that

Similarly,

Undoing the transformation, we can see that ̂ C also satisfies:

i.e. ̂ C diagonalizes S -1 H since ̂ C -1 S -1 H ̂ C = Λ . We conclude that κ V ( S -1 H ) ≤ κ ( ̂ C ) ≤ ∥ L ∥∥ L -1 ∥ = √ κ ( S ) .

A.5 Imported floating point algorithms for fast linear algebra

Before we dive into the details, we recall the concept of 'fast matrix multiplication', pioneered by Strassen [134], who showed that two square matrices can be multiplied in O ( n ω ) , where ω = log 2 7 ≈ 2 . 807 < 3 , in real arithmetic. Since then, the matrix multiplication exponent ω has been significantly reduced, the record to date being ω ≤ 2 . 371552 [47, 141]. In two seminal works [40, 39], Demmel, Dumitriu, Holtz, and Kleinberg, proved that any fast matrix multiplication algorithm can be executed numerically stably in a floating point machine with almost the same arithmetic complexity as in real arithmetic. They also showed that other problems in Numerical Linear Algebra can be reduced to such matrix multiplications, including inversion, LU and QR factorizations, and solving linear systems of equations. Our algorithms build on the existing results for stable fast matrix multiplication [40], inversion [39], as well as backward-approximate diagonalization and the matrix sign function [15]. We import the corresponding results from the aforementioned works in the following theorems.

Theorem A.2 ( MM , stable fast matrix multiplication [40]) . For every η > 0 , there exists a fast matrix multiplication algorithm MM which takes as input two matrices A , B ∈ C n × n and a machine precision u > 0 and returns C ← MM ( A , B ) such that

on floating point machine with precision u , where µ MM ( n ) = n c η , for some constant c η independent of n . Such an algorithm is called µ MM ( n ) -stable, and it requires T MM ( n ) = O ( n ω + η ) arithmetic operations, where ω is the exponent of matrix multiplication in real arithmetic.

Theorem A.3 ( INV , logarithmically-stable fast inversion [39]) . For every η > 0 , there exists a fast inversion algorithm INV which takes as input an invertible matrix A ∈ C n × n and a machine precision u > 0 and returns C ← INV ( A ) such that

on floating point machine with precision u , where µ INV ( n ) = O ( n c η +log(10) ) , for some constants c INV ≤ 8 and c η independent of n . Such an algorithm is called ( µ INV ( n ) , c INV ) -stable, and it requires T INV ( n ) = O ( T MM ( n )) arithmetic operations, where T MM ( n ) is the same as in Theorem A.2.

Sign function and deflation. The next result that we import is a floating point algorithm for the matrix sign function, which, for a diagonalizable matrix A = VΛV -1 such that Λ( A ) does not interesect with the imaginary axis, is defined as sgn( A ) = V sgn( Λ ) V -1 , where sgn( Λ ) is a diagonal matrix and each diagonal entry contains the sign of the real part of the corresponding eigenvalue. To state the main result, we first recall the definition of the Circles of Apollonius .

Definition A.3 (Circles of Apollonius, imported from Section 4.1 of [15]) . Let α ∈ (0 , 1) . The Circles of Apollonius C α are defined as C α = C + α ∪ C -α ,

where m ( z ) = 1 -z 1+ z is the M¨ obius transformation taking the right half-plane to the unit disk. The disk C + α is centered at 1+ α 2 1 -α 2 and has radius 2 α 1 -α 2 , and C -α is its reflection with respect to the imaginary axis.

Theorem A.4 ( SGN , imported Theorem 4.9 from [15]) . There is a deterministic algorithm SGN ( A , α, η, ϵ ) which takes as input a matrix A ∈ C n × n , an accuracy parameter ϵ ∈ (0 , 1 / 12) , and parameters η ∈ (0 , 1) , 0 < 1 -α < 1 / 100 , such that it is guaranteed that Λ η ( A ) ⊂ C α . The algorithm returns a matrix ˜ S such that

as long as the machine precision satisfies

corresponding to at most bits of precision. Here

denotes the number of iterations that the algorithm executes. The arithmetic complexity is

In PCA, we will need to be able to compute a basis for the column space of an approximate spectral projector. For this, we recall the following algorithm.

Theorem A.5 ( DEFLATE , imported Theorem 5.3 of [15]) . There exists a randomized algorithm ˜ C k ← DEFLATE ( ˜ Π k , k, β, ϵ ) , which takes as input a matrix ˜ Π k , a rank parameter k ∈ [ n ] , a parameter β such that ∥ ˜ Π k -Π k ∥ ≤ β , for some projector matrix Π k of rankk , and a desired accuracy ϵ , and returns a (complex) matrix ˜ C k such that, there exists a matrix C k with k columns that forms an orthonormal basis for im( Π k ) , and

The algorithm succeeds with probability at least 1 -(20 n ) 3 √ β ϵ 2 . The number of arithmetic operations is at most O ( T MM ( n )) , and it requires O (log( n )+log( 1 ϵ )) bits of precision. Internally, the algorithm generates O ( n 2 ) random numbers to form a n × n complex Ginibre matrix. 1

Computing an orthonormal basis and the spectral norm. Lastly, we recall the following two results. The first one is for the QR factorization from [39], which we adapt suitably for our analysis, specifically, to compute an orthonormal basis for the column space of a rectangular matrix.

Theorem A.6 (Basis computation, follows from Section 4 in [39]) . Let A ∈ R m × n , m ≥ n . There exists an algorithm ˜ Q , ˜ R = QR ( A ) , which returns a matrix ˜ Q ∈ R m × n and an upper triangular matrix ˜ R ∈ R n × n in O ( mn ω -1 ) floating point operations using O (log( nκ ( A ) ϵ QR )) bits, where ϵ QR ∈ (0 , 1) is a given accuracy. The matrix ˜ Q satisfies

for some orthogonal matrix Φ ∈ R n × n , where Q has orthonormal columns and A = QR is the true economy-QR of A .

Proof. We first scale A ′ ← A /M , where M is the smallest power of two that is larger than n ∥ A ∥ max . This ensures that Ω(1 /n ) ≤ ∥ A ′ ∥ ≤ 1 . Since we scale by a power of 2 , then there are no floating point errors and the orthonormal basis from the QR factorization remains the same (only the upper triangular factor is scaled).

Wenowuse the corresponding algorithm of [39] on A ′ . It returns three matrices: an upper triangular matrix ˜ R ∈ R n × n , a matrix W ∈ R m × n with ∥ W ∥ ∈ O ( n ) , and a matrix Y ∈ R n × m with ∥ Y ∥ ∈ O ( n ) . The matrix Ψ = I -WY ⊤ + E Ψ exactly satisfies ΨΨ ⊤ = I , and the error matrix satisfies ∥ E Ψ ∥ ∈ O (poly( n ) u ) . Moreover, Ψ ( ˜ R 0 m -n × n ) = ̂ A , where ̂ A = A ′ + E A and ∥ E A ∥ ≤

O (poly( n ) u ) ∥ A ′ ∥ ≤ O (poly( n ) u ) . Note that ∥ ˜ R ∥ ≤ ∥ ̂ A ∥ ≤ ∥ A ′ ∥ + ∥ E A ∥ ≤ 1 + O (poly( n ) u ) . The total cost of the algorithm is O ( mn ω -1 ) floating point operations (ignoring the negligible term η > 0 in the exponent).

Using W and Y , we can construct an approximate basis ˜ Q as follows. Let Y n ∈ R n × n contain the first n columns of Y . We can compute the matrix ˜ Q = ( I n 0 m -n × n ) -MM ( W , Y n ) in O ( mn ω -1 ) floating point operations by performing the multiplication in blocks of size n × n . We can write ˜ Q = ̂ Q + E ̂ Q where ̂ Q contains the first n columns of Ψ and ∥ E ̂ Q ∥ ∈ O (poly( n ) u ) .

Note that ˜ R satisfies ̂ Q ˜ R = A ′ + E A . Since ̂ Q has orthonormal columns, the singular values of ˜ R satisfy σ i ( ˜ R ) = σ i ( ̂ Q ˜ R ) = σ i ( A ′ + E A ) ∈ σ i ( A ′ ) ± ∥ E A ∥ , where the last comes from the stability of singular values which is a consequence of Weyl's inequality in Fact A.1. Now, if ∥ E A ∥ ≤ 1 4 σ min ( A ′ ) , which is achieved by setting u ≤ c 1 poly( n ) κ ( A ) for some constant c , then ˜ R is full rank, and it holds that ∥ ˜ R -1 ∥ = 1 /σ min ( ˜ R ) ≤ 4 3 σ min ( A ′ ) ≤ O ( n ) κ ( A ) .

Since ˜ R remains full rank, we have that ̂ Q ˜ R = ̂ A ⇒ ̂ Q = A ′ ˜ R -1 + E A ˜ R -1 . If we write the true economy-QR of A ′ as A ′ = QR , then we can see that ∥ R ˜ R -1 ∥ = ∥ QR ˜ R -1 ∥ = ∥ A ˜ R -1 ∥ =

∥ ̂ Q -E A ˜ R -1 ∥ . This implies that ∥ R ˜ R -1 ∥ ∈ [1 - ∥ E A ˜ R -1 ∥ , 1 + ∥ E A ˜ R -1 ∥ ] . With similar arguments we obtain

We previously argued that ∥ R ∥ , ∥ ˜ R ∥ ≤ O (1) , and ∥ R -1 ∥ , ∥ ˜ R -1 ∥ ≤ O ( n ) κ ( A ) . Thus, if we further enforce ∥ E A ∥ ≤ ϵ 1 c 1 1 poly( n ) κ ( A ) for some constant c 1 , then max {∥ E A ˜ R -1 ∥ , ∥ E A R -1 ∥} ≤ ϵ 1 , for some ϵ 1 ∈ (0 , 1 / 4) , which means that all the singular values of R ˜ R -1 are inside the interval [1 -2 ϵ 1 , 1 + 2 ϵ 1 ] . In other words, R ˜ R -1 is approximately orthogonal. Indeed, we can write R ˜ R -1 = Φ + E Φ where ΦΦ ⊤ = I and all the singular values of E Φ are inside [ -2 ϵ 1 , 2 ϵ 1 ] . In particular, Φ = UV ⊤ , where U , V come from the SVD of R ˜ R -1 = UΣV ⊤ .

We can now go back to ˜ Q . From the above, ̂ Q can be written as

This means that ˜ Q is just a rotation of the true Q plus some additive error terms that we can control with u . In particular,

where in the last we applied the required bound for u ≤ ϵ 1 c 1 1 poly( n ) κ ( A ) .

Combining everything, we conclude that if the machine precision satisfies

then we can compute a matrix ˜ Q in O ( mn ω -1 ) floating point operations such that there exists orthogonal matrix Φ satisfying

using O (log( nκ ( A ) /ϵ QR )) bits of precision.

The second result is for the computation of the spectral norm of the matrix from [107] using the Lanczos algorithm.

Theorem A.7 (Imported variant of Theorem 18 from the full version of [107]) . Let A ∈ R m × n and δ ∈ (0 , 1 / 2) a failure probability parameter. We can compute a vector y such that, with probability at least 1 -δ ,

in O ( mn log( n ) log(1 /δ )) floating point operations using O (log( n )) bits of precision. Internally the algorithm generates O ( n log(1 /δ )) random bits.

We can obtain the following corollary.

Corollary A.1 (Spectral norm) . Let B ∈ C m × n . We can compute a value ˜ Σ ∈ Θ( ∥ B ∥ ) with probability at least 1 -2 δ in O ( mn log( n ) log(1 /δ )) floating point operations using O (log( n )) bits of precision.

Proof. Let R be the real part of B and Z be the imaginary part. We can approximate R ∈ [ 9 10 ∥ R ∥ , ∥ R ∥ ] and Z ∈ [ 9 10 ∥ Z ∥ , ∥ Z ∥ ] using the algorithm of Theorem A.7. We then set ˜ Σ = R + Z ∈ [0 . 9( ∥ R ∥ + ∥ Z ∥ ) , ∥ R ∥ + ∥ Z ∥ ] . Using the triangle inequality we have that

or, in other words, ˜ Σ ∈ [0 . 9 ∥ B ∥ , 2 ∥ B ∥ ] .

which ultimately gives

A.6 Symmetrization

There are certain cases where it is crucial to ensure that the result of a floating point multiplication or inversion remains Hermitian. The following proposition states that we can always 'symmetrize' a floating point matrix with small additional errors.

Proposition A.3 ( HERM ) . Let C be a Hermitian matrix and C ′ = C + E for some (non-Hermitian) matrix E , such that ∥ C -C ′ ∥ = ∥ E ∥ . Consider the matrix HERM ( C ′ ) , where HERM ( A ) is an algorithm that replaces the strictly lower triangular part of A with the strictly upper triangular part of A . Then HERM ( C ′ ) is Hermitian and it holds that:

for some constant c HERM .

Proof. For any matrix A , the upper triangular part of A , denoted as ∆ U ( A ) , satisfies the following inequality:

for some constant c . The same holds for the lower triangular part ∆ L ( A ) . For the diagonal part diag( A )) it holds that ∥ diag( A ) ∥ ≤ ∥ A ∥ (see [27] for proofs). Let C ′′ = HERM ( C ′ ) . By the definition of HERM ( · ) , the matrix C ′′ can be written as C ′′ = ∆ U ( C ′ ) + ∆ U ( C ′ ) ∗ -diag( C ′ ) = C +(∆ U ( E ) + ∆ U ( E ) ∗ -diag( E )) . Therefore

for some constant c HERM .

We can directly use this to derive bounds for matrix multiplication where the result is a Hermitian matrix and for Hermitian matrix inversion.

Corollary A.2 (Symmetrized matrix multiplication) . Let A and B be two matrices and C = AB be a Hermitian matrix. If MM is a µ MM -stable matrix multiplication algorithm then the matrix

is Hermitian and it satisfies

Proof. Straightforward combination of Theorem A.2 and Proposition A.3.

Corollary A.3 (Symmetrized inversion) . Let A be an invertible Hermitian matrix. If INV is a ( µ INV , c INV ) -stable inversion algorithm then the matrix

is Hermitian and it satisfies

In addition, if u satisfies

for some ϵ ∈ (0 , 1 / 2) , then both of the following hold:

Proof. The first part is straightforward combination of Theorem A.3 and Proposition A.3. For the second part, we directly bound

Note that ∥ C -A -1 ∥ ≤ ϵ 1 κ ( A ) ∥ A -1 ∥ = ϵ 1 ∥ A ∥ = ϵσ min ( A -1 ) . Since C and A -1 are both Hermitian then from Weyl's inequality

where we conclude that

Remark A.1. Throughout this paper, to simplify our bounds, we assume that

Moreover, we define a global upper bound µ such that

where, from Theorem A.2, c ′ η is a constant that does not depend on n .

A.7 Existing algorithms for invariant subspaces and the (generalized) eigenproblem

We give a brief overview of some results related to the computation of spectral projectors, eigenvectors, and invariant subspaces for the Hermitian definite generalized eigenproblem. Many more details can be found in standard textbooks and references therein [42, 119, 64, 135, 115], as well as the recent overview of [129] for the complexity of the eigenproblem. For further reading we can highlight some influential works from the enormous bibliography on eigenvalue algorithms and perturbation bounds [44, 43, 8, 130, 131, 132, 133, 75, 108, 9, 26, 76, 77, 37, 2, 113, 112].

The prominent classic method to solve the Hermitian (generalized or regular) eigenproblem is to use numerically stable Householder transformations to reduce a Hermitian matrix to tridiagonal form, and then apply a tridiagonal shifted QR algorithm to diagonalize the tridiagonal matrix, which gives both the eigenvalues and the eigenvectors. The paramount QR algorithm was originally proposed in [51, 52, 90], its convergence for the symmetric case was analyzed in [140, 38] in exact arithmetic, and the non-symmetric was recently analyzed in floating point in [18, 16, 17]. Other classical algorithms for computing invariant subspaces include [97, 98, 10, 9].

With the aforementioned procedure, the floating point complexity is about ˜ O ( n 3 ) floating point operations, potentially up to polylog(1 /ϵ, 1 / gap k ) factors. It is known that for Hermitian definite pencils ( H , S ) one can extend this procedure such that, simultanteously, H will become (Hermitian) tridiagonal and S the identity [36]. The Hermitian tridiagonal matrix can be further reduced with similarity transformations to a symmetric tridiagonal matrix, in which case one can apply again tridiagonal symmetric QR. Using standard perturbation bounds as the ones used in this work one can potentially obtain provable forward errors invariant subspaces.

While there exist works that overcome the ˜ O ( n 3 ) worst-case complexity barrier for some computations related to the eigenproblem, none of them provide end-to-end forward errors for individual invariant subspaces. [114] showed that the eigenvalues of a matrix can be computed in O ( n ω ) arithmetic operations, however, it becomes O ( n ω +1 ) boolean operations in rational arithmetic (up to some other omitted factors), and therefore slower than standard ˜ O ( n 3 ) floating point eigensolvers. [25] proposed a quantum-inspired method for approximate diagonalization in a backward-error sense, with O ( n ω +1 ) bit complexity. [15] proved that backward-approximate diagonalization can be solved in ˜ O ( n ω ) bit complexity in floating point by using smoothed analysis, improving both the ˜ O ( n 3 ) classic algorithms for the Hermitian case and the O (poly( n/ϵ )) algorithm of [7] for

the non-Hermitian case. This was extended for the generalized eigenproblem case in [41]. As already mentioned in the introduction, to obtain forward errors for eigenvectors from the backwardapproximate solution, the corresponding bounds that are reported in [15, 41] require the existence of a minimum eigenvalue gap in the original matrix, while the analysis for invariant subspaces was left as an open problem. Our work completes this analysis, and takes a step even further, by showing that (at least based on existing results), explicit diagonalization is redundant for approximating individual invariant subspaces.

B Proof of Proposition 2.1

PURIFY Input: Hermitian definite pencil H ∈ H n , S ∈ H n ++ , approximate condition number ˜ κ of S , approximate values ˜ µ k , ˜ gap k , accuracy ϵ ∈ (0 , 1 / 12) . Requires: ∥ H ∥ ≤ 1 , ∥ S -1 ∥ ≤ 1 , ˜ κ ∈ Θ( κ ( S )) , ˜ µ k ∈ µ k ± 1 8 gap k , and ˜ gap k ∈ (1 ± 1 8 ) gap k , where µ k = λ k + λ k +1 2 and gap k = λ k -λ k +1 for some k ∈ [ n -1] . Algorithm: ˜ P ← PURIFY ( H , S , ˜ µ k , ˜ gap k , ˜ κ, ϵ ) . 1: S INV ← INV ( S ) . ▷ S INV = S -1 + E INV 1 2: ˜ H ← MM ( S INV , H ) . ▷ ˜ H = S INV H + E MM 2 3: ˜ M ← ˜ µ k -˜ H + E ( -) 3 . 4: ˜ C ← SGN ( ˜ M , ϵ 16 · 199 , ˜ gap k 32 ) . ▷ ˜ C = sgn( ˜ M ) + E SGN 4 5: ˜ Π ← 1 2 ( 1 + ˜ C ) + E (+) 5 6: return ˜ Π . Output: Approximate spectral projector ˜ Π . Ensures: ∥ ˜ Π -Π ∥ ≤ ϵ .

For the analysis of Algorithm 3 we first recall that, in exact arithmetic, small perturbations do not have a severe influence in the sign function of a matrix.

Lemma B.1. Let H ∈ H n , S ∈ H n ++ , where µ is a scalar, and M = µ -S -1 H and ˜ M = µ -S -1 H + E . If ∥ E ∥ ≤ ϵ | λ min ( M ) | 2 π 128 κ V ( M ) for some ϵ ∈ (0 , 1) , where | λ min ( M ) | is the smallest eigenvalue of M in absolute value, then ∥ sgn( M ) -sgn( ˜ M ) ∥ ≤ ϵ .

Proof. The proof uses standard techniques from holomorphic functional calculus and the properties of the pseudospectrum, but it is stated for completeness. A similar proof, for example, can be found in [9] for general matrices A (i.e. in that proof A can have complex eigenvalues).

Recall that for a matrix A that has no eigenvalues on the imaginary axis, it holds that

where P -( A ) is the spectral projector on the subspace spanned by the eigenvectors corresponding to eigenvalues with negative real part, and P + ( A ) is the corresponding spectral projector to the positive halfplane. Then

S -1 H has real eigenvalues because it is similar to a Hermitian matrix, i.e. Λ( S -1 H ) = Λ( L ∗ HL ) where L is the lower triangular Cholesky factor of S -1 . Thus M has also real eigenvalues. From Proposition A.1 (i) and (iii)

If ∥ E ∥ ≤ ϵ 1 | λ min ( M ) | 4 κ V ( M ) for some ϵ 1 ∈ (0 , 1) then | λ min ( ˜ M ) | ≥ | λ min ( M ) | / 2 and | λ max ( ˜ M ) | ≤ | λ max ( M ) | + | λ min ( M ) | / 2 .

Let R be a rectangle whose bottom-left corner is located at -i | λ min ( M ) | and its top-right corner at 2 + | λ min ( M ) | + i | λ min ( M ) | . Any point on the boundary of R has a distance of at least | λ min ( M ) | from Λ( M ) and at least at least | λ min ( M ) | / 2 from Λ( ˜ M ) . From holomorphic functional calculus we have that

where in this case ∂R denotes a positively oriented rectifiable curve over the boundary of the rectangle. Using the resolvent identity ( z -M ) -1 -( z -M ′ ) -1 = ( z -M ) -1 ( M -M ′ )( z -M ′ ) -1 , for P + we have

Since | λ min ( M ) | ≤ 2 , then if we set ϵ 1 ≤ ϵ | λ min ( M ) | π 16 (which satisfies ϵ 1 < 1 ), where ϵ is the desired final accuracy, this gives

The same can be obtained for P -. Putting everything together, if ∥ E ∥ ≤ ϵ | λ min ( M ) | 2 π 128 κ V ( M ) for some ϵ ∈ (0 , 1) ensures that ∥ sgn( M + E ) -sgn( M ) ∥ ≤ ϵ . From Proposition A.2 κ V ( M ) ≤ √ κ ( S ) , which gives the final bound.

We can now prove Proposition 2.1, which we restate for readability.

Proposition B.1 (Restatement of Proposition 2.1) . Let H ∈ H n with ∥ H ∥ ≤ 1 , S ∈ H n ++ with ∥ S -1 ∥ ≤ 1 , k ∈ [ n -1] and ϵ ∈ (0 , 1) . Let µ k = λ k + λ k +1 2 and gap k = λ k -λ k +1 , where λ 1 ≤ . . . ≤ λ n are the generalized eigenvalues of the Hermitian definite pencil ( H , S ) and assume that we want to compute Π k which is the true spectral projector associated with the k smallest eigenvalues. If we have access to

for some constant C > 1 , then Algorithm 3 computes ˜ Π k ← PURIFY ( H , S , ˜ µ k , ˜ gap k , ˜ κ, ϵ ) such that ∥ ˜ Π k -Π k ∥ ≤ ϵ, in O ( T MM ( n ) ( log( 1 gap k ) + log(log( κ ( S ) ϵ gap k )) )) floating point operations using O ( log( n ) log 3 ( 1 gap k ) log( κ ( S ) ϵ gap k ) ) bits of precision.

Proof. Using the notation of Algorithm 3, we have that ˜ H = S -1 H + E INV 1 H + E MM 2 . Let us initially set

for some ϵ 0 ∈ (0 , 1 / 8) (to be determined later). From Theorems A.3 and A.2, and from the assumption that ∥ H ∥ , ∥ S -1 ∥ ≤ 1 we have that

We next have ˜ M ← ˜ µ k -˜ H + E ( -) 3 = ˜ µ k -S -1 H + E INV 1 H + E MM 2 + E ( -) 3 = ˜ µ k -S -1 H + B , where E ( -) 3 is a diagonal error matrix with

and B = E INV 1 H + E MM 2 + E ( -) 3 . We can bound the norm of B as:

Now we need to apply Lemma B.1 to argue that ∥ ∥ sgn( µ k -S -1 H ) -sgn( ˜ µ k -S -1 H + B ) ∥ ∥ ≤ ϵ 1 , for some ϵ 1 ∈ (0 , 1) . To satisfy the requirements of the lemma, we need to ensure that ∥ B ∥ ≤ ϵ 1 π | λ 2 min ( ˜ µ k -S -1 H ) | 128 √ κ ( S ) . This can be achieved by setting

However, we do not know | λ min ( ˜ µ k -S -1 H ) | and κ ( S ) . To circumvent this, we can use the assumption that ˜ µ k is well-placed close to the center of gap k , which implies that

and also use the assumption that κ ( S ) ≤ ˜ κ . This means that we can set

which implies the desired bound for B :

We can now apply Lemma B.1 to argue that

By assumption we know that ˜ µ k = µ k ± 1 8 gap k which means that sgn( ˜ µ k -S -1 H ) = sgn( µ k -S -1 H ) , and therefore

To not interrupt the flow, we will skip for now the analysis of SGN and we will leave it for the end of the proof. From Theorem A.4 we know that the result returned by SGN satisfies ∥ ˜ C -sgn( ˜ M ) ∥ ≤ ϵ SGN , where we have full control over ϵ SGN since it is passed as an argument to SGN , and therefore it can be set to ϵ SGN = ϵ 1 . Denoting C = sgn( µ k -S -1 H ) and E C = ˜ C -C , we have

We now proceed to the line 5 of Algorithm 3, which computes ˜ Π ← 1 2 ( 1 + ˜ C ) + E (+) 5 , where E (+) 5 is a diagonal error matrix with norm bounded by

Then we can write ˜ Π = 1 2 (1 + C + E C ) + E (+) 5 = Π + 1 2 E C + E (+) 5 = Π + E Π , where Π = 1 2 (1 + C ) and E Π = 1 2 E C + E (+) 5 . Combining with the above:

in which case we can set ϵ 1 = ϵ 4 , where ϵ is the desired accuracy, to guarantee forward error of at most ϵ for the spectral projector ˜ Π .

Gathering all the requirements for the machine precision (except for SGN , which is detailed below) from Equations (6) and (7), and the assumption on ˜ κ and ˜ gap k it suffices to set

which translates to

bits of precision.

We finally proceed with the analysis of SGN . To use Theorem A.4 for the convergence of SGN , we need to find the appropriate parameters α SGN , η SGN to call it. These parameters must be such that Λ η SGN ( ˜ M ) ⊆ C α SGN , and 99 100 < α SGN < 1 . From the properties of the pseudospectrum, specifically, from Proposition A.1 ( i ) , we know that for any η > 0

where B max is the upper bound for ∥ B ∥ that we obtained above:

For η SGN = B max and from Proposition A.1 ( iii ) we obtain

where we used the fact that κ V ( S -1 H ) ≤ √ κ ( S ) . We can get a rough upper bound

which gives

It suffices to find the appropriate α SGN such that the Apollonian circles C α SGN will contain the disks of the pseudospectrum. Since the smallest singular value of ˜ µ k -S -1 H is lower bounded by 3 8 gap k , the leftmost positive point where any pseudospectral disk intersects with the real axis is 3 8 gap k -ϵ 1 gap 2 k 128 ≥ gap k ( 3 8 -1 128 ) = gap k 47 128 > ˜ gap k 4 . Similarly for the negative halfplane, the rightmost negative point where a disk intersects with the real axis is -˜ gap k 4 . Denoting by ζ = ˜ gap k 8 ,

it suffices to set α SGN = 1 -ζ 1+ ζ = 8 -˜ gap k 8+ ˜ gap k . Clearly, α SGN < 1 for all ∞ > gap k > 0 . For the lower bound, to ensure α SGN > 99 / 100 we need that ˜ gap k < 8 199 . To achieve this we can simply scale the input matrices by a constant, since, by assumption, ∥ S -1 H ∥ ≤ 1 , and therefore gap k ≤ 2 . To conclude, we have all the required parameters to call SGN .

We can now go back to the complexity analysis. We are running

where

We now invoke Theorem A.4. The number of iterations is bounded by

The number of required bits is

which dominates the previous bit bound of Equation 8.

It remains to bound the arithmetic complexity. There is a constant number of inversions, matrix multiplications, and scalar-matrix operations, which in total take at most O ( T MM ( n )) floating point operations. Thus, the dominant factor is the call to SGN , which amounts to

arithetic operations using the aforementioned number of bits.

B.1 Spectral gaps with diagonalization

The straightforward approach to compute the desired spectral gaps is to iteratively compute S -1 H and diagonalize it, until the eigenvalues are well approximated. If we compute A ← MM ( INV ( S ) , H ) , then we can write A = S -1 H + E , for some error matrix E . The next step is to approximate gap k ( S -1 H ) and µ k ( S -1 H ) . For this one could use the recent state-of-the-art backward-approximate diagonalization algorithm of [15].

Theorem B.1 ( EIG , imported Theorem 1.6 from [15]) . There exists a randomized algorithm EIG ( A , ϵ EIG ) which takes any matrix A ∈ C n × n with ∥ A ∥ ≤ 1 and a desired accuracy parameter ϵ EIG > 0 as inputs and returns a diagonal D and an invertible matrix V such that

in

arithmetic operations on a floating point machine with

bits of precision, with probability at least 1 -14 /n .

Applying EIG on A we can obtain a backward-approximate diagonalization. But we are not finished yet, since we are interested in each individual eigenvalue. To translate the backward error to a forward error for the eigenvalues, and, ultimately, the spectral gap, one can try to use Corollary 1.7 and Proposition 1.1 of [15]. However, this approach has two main limitations. First, it relies on simplicity of the spectrum, i.e., it assumes that the minimum gap between any pair of eigenvalues is larger than zero. This assumption is quite restrictive, since the desired gap might be well-defined even at the presence of other multiple eigenvalues. For example, in DFT applications it is not uncommon to have eigenenergies with algebraic multiplicity larger than one, and at the same time have a large band-gap that separates the occupied from the unoccupied orbitals. The second limitation is that the aforementioned Corollary 1.7 requires as an input parameter an actual over-estimate for the minimum eigenvalue gap. Even if such a gap exists, it is not described how to estimate it.

For diagonalizable matrices, we can leverage the following Corollary B.1, which is an immediate consequence of Kahan's inequality (Fact A.1), and it overcomes the aforementioned limitations.

Corollary B.1. If X is diagonalizable and it has real eigenvalues then for any Z the following bound holds for the eigenvalues of X :

Proof. Write X = WΛW -1 where W diagonalizes X and is chosen such that κ ( W ) = κ V ( X ) . This is always possible since if W is any matrix that diagonalizes X then Λ is similar to X and since X has real eigenvalues then Λ has to be real, and therefore symmetric. Then we can write

Since Λ is symmetric, and since W -1 ZW is similar to Z then from Kahan's inequality (Fact A.1)

Now we need to use this to get a bound for the computed generalized eigenvalues after INV and EIG . Proposition B.2 ( EIG -based gap) . Given a definite pencil ( H , S ) with ∥ H ∥ , ∥ S -1 ∥ ≤ 1 , we can compute ˜ gap k ∈ (1 ± ϵ ) gap k and ˜ µ k ∈ µ k ± ϵ gap k by iteratively calling INV , MM , and EIG , using

bits of precision with probability at least 1 -O ( log(1 /ϵ gap k ) n ) .

Proof. Let D , V ← EIG ( A , ϵ EIG ) be the solution returned by EIG when applied to A = S -1 H + E . Note that

From Corollary B.1, since S -1 H is diagonalizable with real eigenvalues then we conclude that

We can tune the machine precision such that ∥ E ∥ = ϵ EIG = ϵ ′ 1 c log( n ) √ κ ( S ) for some chosen ϵ ′ , and some global constant c , then finally

floating point operations using

We can now consider an iterative scheme, where we call EIG on S -1 H , halving ϵ ′ at each step. We need to keep halving ϵ ′ until it reaches ϵ ′ = Θ( ϵ gap k ) , in which case we have a total of O (log( 1 ϵ gap k )) calls to EIG . In the worst case every call costs

arithmetic operations using O ( log 4 ( nκ ( S ) ϵ gap k ) log( n ) ) bits (note that the bits required by MM and INV to achieve accuracy ϵ ′ are dominated by those required for EIG , and we therefore ignore them). Since we do not know gap k , we can set the termination criterion to be ϵ ′ ≈ Θ( ϵ ˜ gap k ) , where ˜ gap k is the approximate gap that we obtain from EIG .

Each iteration succeeds with high probability 1 -1 /n , in which case a union bound gives 1 -O (log(1 / ( ϵ gap k )) /n , which can potentially be improved to 1 -O (1 /n ) without impacting the complexity, but we do not expand further.

A subtle detail in the analysis above is that we need an estimate for κ ( S ) in order to be able to use INV and to set the machine precision. Since κ ( S ) is generally unknown, we need to compute it. We will later show in Appendix E how to compute the condition number quickly.

When S = I , we can avoid the inversion and the computation of κ ( S ) . The arithmetic complexity and the bit requirement are the same by setting κ ( S ) = 1 .

Remark B.1. We highlight that a similar result can be obtained by using the more recent pencil diagonalization algorithm of [41]. The authors the latter mention that their algorithm should be 'more numerically stable' than EIG of [15], since it uses an inverse-free iteration internally, and they do provide strong theoretical and experimental evidence for this statement. However, as there is no formal, end-to-end proof for the bit complexity at the time of this writing, we choose to compare against EIG of [15]. Note that, any improvements on the bit requirements for the matrix sign function, directly provide the same improvements for our main Theorems 1.1 and 3.1.

C Analysis of Cholesky

This section is devoted to the analysis of Algorithm 2 and the proof of Theorem 1.2. Let M = ( A B ∗ B C ) be a Hermitian matrix. It can be factorized in the form

where S = C -BA -1 B ∗ is the Schur complement and

Proposition C.1. Let M ∈ H n ++ and consider the partitioning M = ( A B ∗ B C ) . The following hold

(i) A , C and the Schur complement S = C -BA -1 B ∗ are all positive definite;

(ii) For all X ∈ { A , C , S } we have that ∥ X ∥ ≤ ∥ M ∥ and ∥ X -1 ∥ ≤ ∥ M -1 ∥ ;

(iii) ∥ B ∥ ≤ ∥ M ∥ / 2 .

Proof. It is easy to see that A is positive definite: since M ≻ 0 , then it must also hold that A ≻ 0 since the quadratic form x ∗ Ax can be written as y ∗ My for some vector y . For the Schur complement we recall the factorized form of Equation (9). Consider the quadratic form

means that every quadratic form for the Schur complement can be written as a quadratic form for the matrix M , and therefore they are both positive definite.

For the spectral norm bounds, since A and C are both positive definite then their norms are equal to the largest absolute eigenvalues. From the variational characterizaiton of eigenvalues and the discussion above it is easy to see that ∥ A ∥ ≤ ∥ M ∥ . For the Schur complement, let z be the eigenvector such that ∥ C -BA -1 B ∗ ∥ = z ∗ ( C -BA -1 B ∗ ) z . Then

where we used the fact that BA -1 B ∗ is positive semi-definite (since A -1 is positive definite) and therefore for all x

where equality with zero holds only when z ∈ ker( B ∗ ) .

We finally prove the bound for ∥ B ∥ . Let u be the top left singular vector of B and v be the top right singular vector. Specifically, if B = UΣV ∗ is the economy SVD of B , then u is the first column of U and v ∗ is the first row of V ∗ . Then u ∗ Bv = σ max ( B ) v ∗ v = ( B ) u ∗ u = ∥ B ∥ . Consider the

where we used the fact that A , C are positive-definite. We can then obtain bounds for ∥ C -1 ∥ , ∥ A -1 ∥ , and ∥ S -1 ∥ , by observing that σ min ( X ) = min ∥ z ∥ =1 z ∗ Xz and lower bound it by σ min ( M ) using similar argumnents.

Since A and S are Hermitian and positive-definite, we can recursively compute their Cholesky factors. This gives rise to the recursive Algorithm 2. The complexity is as follows.

Proposition C.2. Algorithm 2 requires O ( T MM ( n )) arithmetic operations.

Proof. Let T ( n ) be the number of operations executed for a matrix of size n × n . Steps 4 and 8 require time T ( n/ 2) . Step 5 requires time T MM ( n/ 2) + T INV ( n/ 2) , and step 6 requires T MM ( n/ 2) . The Schur complement in Step 7 requires T MM ( n/ 2) + T HERM ( n/ 2) + ( n/ 2) 2 = T MM ( n/ 2) + 2( n/ 2) 2 . This becomes

C.1 Error analysis

The goal is to show that CHOLESKY ( M ) will return a backward-approximate Cholesky factor L such that the error

where f is some function, and use this bound to argue about the number of bits that are required to achieve it. Recall that

It is easy to see that

which means that it suffices to bound the individual terms in the sum. The first term is the error of the first recursive call. Following the notation of [39], let us denote err ( n ) the norm-wise error of the algorithm for size n . We can then write

To simplify the various inequalities in the proofs we use the global bound µ from Remark A.1, where all the terms µ MM ( n ) and µ INV ( n ) are bounded by µ ( n ) . Finally, we also define

We can now get a first expression for the bounds of the terms in Equation (10).

Lemma C.1. In Algorithm 2, the blocks L 11 , L 21 , L 22 of the returned matrix L satisfy:

Proof. The error for L 11 L ∗ 11 was already described. For the second inequality, we first expand L 21 L ∗ 11 as follows:

Then

With a similar procedure we can derive the third bound. We first write L 21 = BA -1 L 11 + E L 21 where E L 21 contains several error matrices from previous computations. Then

Similarly

Then

With this, the task reduces to finding appropriate bounds for the norms of the matrices E MM 3 , E BA i , E S and E L 21 . We first derive bounds for the norms of the error matrices E ( · ) .

Lemma C.2. In Algorithm 2, the following bounds hold for the error matrices

Proof. Each term is bounded as follows.

(i) For ∥ E INV ∥

(ii) Similarly, for ∥ E MM 2 ∥ :

(iii) Using these two inequalities we can bound the norm of E BA i :

(iv) Next is the error term of the matrix multiplication between the result of MM ( B , INV ( A )) and L 11 . Expanding Theorem A.2 gives

It suffices to bound ∥ BA i ∥ = ∥ ∥ ∥ B ( A -1 + E INV 1 ) + E MM 2 ∥ ∥ ∥ ≤ ∥ BA -1 ∥ + ∥ BE INV 1 ∥ + ∥ E MM 2 ∥ . Using Proposition C.1, the triangle inequality, and the previous bounds, the two

unknown norms in the sum are bounded as follows

This gives

where in the last inequality we simplified ( 1 + u · E 1 ( n 2 , M )) ≤ ( 1 + u · E INV ( n 2 , M )) . Then

concluding the bound for E MM 3

(v) The norm of E MM 4 is very similar to that of E MM 3 since they both involve a multiplication with BA i :

(vi) Next is E SUB 5 . From Equation (5):

(vii) The next term is E L 21 , which we can bound as follows:

(viii) The final and most involved error term is the norm of E S . Recall that from Line 7 of Algorithm 2 the matrix can be written as

Once more, the norm of each term is bounded separately.

Similarly,

The final bound for E S is given by the sum of the four bounds that were derived.

C.2 Maintaining positive-definiteness, norms, and condition numbers throughout the recursion

Given the bounds of Lemma C.2, we can now calculate the appropriate machine precision u and the corresponding number of bits such that the algorithm will not break down due to loss of positivedefiniteness of the submatrices. Hereafter, we will denote by M the matrix that is passed as an argument in any of the recursive calls of Algorithm 2, and M 0 will denote the original matrix that needs to be factorized.

Lemma C.3. Let M 0 ∈ H n ++ be a matrix that is factorized using Algorithm 2 and κ be its condition number. Then there exist constants c 1 , c 2 , c 3 , c 4 ≥ 1 , such that if n > c 1 and

then every matrix that is constructed in Line 7 during the recursion, that is, every Schur complement ˜ S and every upper-left block A , will be Hermitian and positive-definite. Moreover, for each such matrix X it holds that ∥ X ∥ ≤ 2 ∥ M 0 ∥ and κ ( X ) ≤ 2 κ ( M 0 ) . This value of u translates to

required bits of precision.

Proof. At each recursive step, two matrices need to remain positive-definite: A and ˜ S . If M is positive-definite then so is A . It remains to ensure the same for the Schur complement. To prove this we first fix u be bounded by the value

Assume that we are in the first level of recursion, i.e. M = M 0 . Per Lemma C.2, we can write ˜ S = S + E S , where

For the assumed value of u , both u · E 1 ( n 2 , M 0 ) ≤ 1 and u · E INV ( n 2 , M 0 ) ≤ 1 , and we also know that 1 ≤ κ ( M 0 ) , in which case the bound simplifies to

Using this, we derive the following useful inequalities

In the above, we used the fact that for c INV ≥ 1 we have that n 2 c INV +1 n 2 c INV -1 ≤ 2 , for all n ≥ 3 , and therefore 2 c INV log( n/ 4) = ( n/ 4) c INV ≤ n c INV . Since the smallest eigenvalue of ˜ S is larger than a positive value, we can conclude that it is Hermitian and positive-definite, and its condition number and spectral norm are appropriately bounded.

In a similar manner, we can now get a bound for the Schur complement of the Schur complement. In this recursive step we call Algorithm 2 with M = ˜ S . Let

Let S 1 be the true Schur complement of ˜ S , i.e. S 1 = C 1 -B 1 A -1 1 B ∗ 1 , and let ˜ S 1 be the approximate Schur complement of ˜ S that is constructed in line 7 of Algorithm 2 when executed on ˜ S . Let E S 1 be the corresponding error matrix, i.e. ˜ S 1 = S 1 + E S 1 . Using Lemma C.2 and simplifying the terms (1 + 1 /n c INV ) ≤ 2 and n 2 c INV +1 n 2 c INV -1 ≤ 2 we have that

The corresponding bounds for ˜ S 1 can be derived:

We can therefore identify that ˜ S 1 is also Hermitian positive-definite and its extremal eigenvalues are similarly bounded with those of ˜ S .

If we keep applying the same analysis for all log( n ) iterations, as long as n is greater than some constant such that ( n c INV +log n ) / ( n c INV -log n ) ≤ 2 , then for each i = 1 , ..., log n it holds that

which implies that

The same bounds hold for the matrices A i , since they always originate from the top-left corner of a matrix ˜ S j , where j < i , i.e., A i either has no errors or it inherits the errors from a matrix ˜ S j . We can therefore conclude that, for the value of u in Inequality (13), every matrix A i and ˜ S i that is constructed during the recursion of Algorithm 2 will be Hermitian and positive-definite and its condition number and spectral norm will be at most 2 κ ( M 0 ) and 2 ∥ M 0 ∥ respectively. The required number of bits is

C.3 Final backward-approximation bounds and proof of Theorem 1.2

Having safeguarded the possibility of a breakdown due to loss of positive-definiteness, we can now revisit the bounds of Lemma C.1 and finalize the proof of Theorem 1.2.

Theorem C.1 (Restatement of Theorem 1.2) . Given a Hermitian positive-definite matrix M , there exists an algorithm L ← CHOLESKY ( M ) , listed in Algorithm 2, which requires O ( T MM ( n )) arithmetic operations. This algorithm is logarithmically stable, in a sense that, there exist global constants c 1 , c 2 , c 3 , such that for all ϵ ∈ (0 , 1) , if executed in a floating point machine with precision

which translates into O ( log( n ) log( κ ( M )) + log( 1 ϵ ) ) required bits of precision, then it does not break down due to arithmetic errors, and the solution returned satisfies ∥ LL ∗ -M ∥ ≤ ϵ ∥ M ∥ .

Proof. The arithmetic complexity of the algorithm was already bounded in Proposition C.2.

For the error proof we assume the bound for u ≤ u ++ from Inequality (13). Combining the inequalities for a single recursion step from Lemma C.2 with the results of the previous section and the bound for u , we have the following inequalities (recall once more that M is the input of the algorithm for the specific recursive call, while M 0 is the matrix in the original call):

Using those inequalities, the first term that we need to bound from Lemma C.1 is the following

where in the last inequality we used the assumed bound for u , namely Inequality (13). Similarly

where in the last inequality we used again (13).

2

We can now derive a recursive formula for the main error bounds, Inequality (10).

The base case n = 1 is trivial since M is just a positive real number 2 and we can compute L = fl ( √ M ) = (1 + δ ) √ M with | δ | ≤ u , which means that LL ∗ = L 2 = (1 + δ ) 2 M > 0 and

If we denote α = 196 κ ( M 0 ) 2 and β = u · 422 κ ( M 0 ) 2 n c INV E INV ( n 2 , M 0 ) , then the solution of the recursion can be written as

We can now absorb the remaining terms inside u , by tuning the constants of Inequality (13), to argue that we can achieve any desired (multiplicative) backward-accuracy ϵ ∈ (0 , 1) and if we set

for some constants c ′ 1 , c ′ 2 , c ′ 3 . This translates to the advertised

bits of precision.

C.4 Reducing the GEP to a regular Hermitian eigenproblem

We can now use CHOLESKY to reduce the definite GEP to a regular Hermitian eigenproblem For the rest of the paper, we assume that ∥ H ∥ , ∥ S -1 ∥ , ∥ S -1 H ∥ ≤ 1 . This is not a limitation since we can approximate the norms of H and S -1 in floating point using the algorithm SIGMAK , which is described later, and then scale accordingly. Formally, let η ≳ ∥ H ∥ and σ ≳ ∥ S -1 ∥ . Then we can rewrite the generalized eigenproblem

i.e. it is the same generalized eigenproblem only with scaled eigenvalues. We can thus safely make the unit-norms assumption. In Proposition C.3 we prove the properties of the reduction, which, in brief, states that we can compute in O ( T MM ( n )) floating point operations using O (log( n ) log( κ ( S )) + log(1 /ϵ )) bits the matrix ˜ H ← ˜ L ∗ H ˜ L , , where ˜ L is returned by CHOLESKY ( INV ( S )) . The eigenvalues of ˜ H provably approximate the pencil eigenvalues:

where ϵ ∈ (0 , 1) is a given accuracy. Algorithm 4 details the reduction.

REDUCE Input: Matrix H ∈ H n , matrix S ∈ H n ++ , accuracy parameter ϵ ∈ (0 , 1) . Requires: H is Hermitian and ∥ H ∥ ≤ 1 , S is Hermitian and positive definite and ∥ S -1 ∥ ≤ 1 . Algorithm: ˜ H ← REDUCE ( H , S ) . 1: S INV ← HERM ( INV ( S )) . ▷ S INV = S -1 + E INV 1 . 2: ˜ L ← CHOLESKY ( S INV ) . ▷ ˜ L ˜ L ∗ = S INV + E CH 2 . 3: ˜ H ← HERM ( MM ( MM ( ˜ L ∗ , H ) , ˜ L )) . ▷ ˜ H = ( ˜ L ∗ H + E MM 3 ) ˜ L + E MM 4 = ˜ L ∗ H ˜ L + E 5 . 4: return ˜ H . Output: Hermitian matrix ˜ H . Ensures: | λ i ( ˜ H ) -λ i ( S -1 H ) | ≤ ϵ in O ( T MM ( n )) floating point operations using O (log( n ) log( κ ( S )) + log(1 /ϵ )) bits.

Algorithm 4: REDUCE .

Proposition C.3. Given H ∈ H n , S ∈ H n ++ , and ϵ ∈ (0 , 1) , Algorithm 4 executes O ( T MM ( n )) floating point operations and returns a matrix ˜ H ← REDUCE ( H , S , ϵ ) , such that, if the machine precision satisfies

for some constants ρ 1 , ρ 2 , ρ 3 , and ϵ ∈ (0 , 1) , which translates to

required bits of precision, then for all i ∈ [ n ]

where λ i ( H , S ) are the eigenvalues of the Hermitian definite pencil ( H , S ) .

Proof. Let L be the (exact) lower triangular Cholesky factor of S -1 , i.e. LL ∗ = S -1 (which is unique up to column phases). Expanding the equation in line 3 of Algorithm 4 we have

At the same time, Λ( S -1 H ) = Λ( L ∗ HL ) . Combining this with Kahan's bound from Fact A.1 and denoting

we have that

for some constant C . It remains to bound for ∥ B ∥ .

(i) E INV 1 originates from the inversion of S INV . Let S INV = HERM ( INV ( S )) = S -1 + E INV 1 . From Corollary A.3, we know that as long as the machine precision satisfies

for some δ ∈ (0 , 1 / 2) , then all of the following hold: ∥ E INV 1 ∥ ≤ δ ∥ S -1 ∥ , 1 2 ∥ S -1 ∥ ≤ ∥ S INV ∥ ≤ 2 ∥ S -1 ∥ , 1 4 κ ( S ) ≤ κ ( S INV ) ≤ 4 κ ( S ) .

(ii) As a second step, the Cholesky factor ˜ L = CHOLESKY ( S INV ) is computed, such that ˜ L ˜ L ∗ = S INV + E CH 2 . From Theorem 1.2 we know that as long as

for some constants c 1 , c 2 , c 3 , then ∥ E CH 2 ∥ ≤ δ ∥ S INV ∥ ≤ 2 δ ∥ S -1 ∥ . From the bound of κ ( S INV ) in ( i ) , the following is sufficient:

(iii) Finally, we form the matrix ˜ H = HERM ( MM ( MM ( ˜ L , H ) , ˜ L ∗ ) ) = ˜ L ∗ H ˜ L + E MM 3 ˜ L + E MM 4 = E 3 . Using Theorem A.2 and Corollary A.2, as long as

(iv) For the final desired error we need to bound ∥ ˜ L ∥∥ ˜ L -1 ∥ . Taking the square:

For the spectral norm we have that ∥ S INV + E CH 2 ∥ ≤ ∥ S INV ∥ + ∥ E CH 2 ∥ ≤ 2 ∥ S -1 ∥ +2 δ ∥ S -1 ∥ ≤ 3 ∥ S -1 ∥ . For the spectral norm of the inverse we need to bound the smallest singular value of

Combining the last two bounds, that we have

We now have bounds for all the required quantities to bound ∥ B ∥ :

Setting δ = ϵ 1 64 C log( n ) κ ( S ) , where C is the constant from Inequality (17) and ϵ ∈ (0 , 1) , this finally gives

which we can use in Inequality (17) to obtain

We can now gather all the requirements for u REDUCE . From the above, u REDUCE needs to satisfy

for some suitably chosen constants ρ 1 , ρ 2 , ρ 3 . This translates to

required bits of precision. Having established the eigenvalue bounds, note that λ i ( H , S ) = λ i ( L ∗ HL , I ) . Since ˜ H and L ∗ HL are both Hermitian, the eigenvalue bound holds also for the singular values. For scaled matrices ∥ S -1 ∥ , ∥ H ∥ ≤ 1 we obtain a maximum ϵ additive error for the eigenvalues.

C.5 Counterexample for floating point LU

In this section provide a counter-example which illustrates why the backward-stable LU factorization algorithm of [39] is not sufficient to obtain a Cholesky factor for a Hermitian postive-definite matrix. We apply one-by-one the steps of the LUR algorithm of [39] on the matrix:

(a) In the first step the matrices L L and U L are computed by calling LUR ( A 1:2 , 1 ) , to compute the LU factorization of the first column of A . It is easy to see that this returns L L = ( 1 1 3 (1 + θ 1 ) ) where | θ 1 | ≤ u , and U L = 3 (the U L factor is a 1-by-1 matrix in this case).

(b) The second step updates the upper right corner of A , by multiplying from the left with the inverse of the top-left block of L L . Assuming that multiplication and inversion with the number 1 does not incur errors, then for this specific example A 1 , 2 is not modified in this step, i.e., it remains equal to 1 , without errors.

(c) The third step updates the Schur complement, which becomes A 2 , 2 ← fl ( A 2 , 2 -1 3 (1 + θ 1 )) = (3 -1 3 (1 + θ 1 ))(1 + θ 2 ) , where | θ 2 | ≤ u is the error term incurred by subtraction.

(d) The next step computes the L R , U R ← LUR ( A 2 , 2 ) of the (updated) Schur complement. In this specific case it trivially returns L R = 1 and U R = A 2 , 2 = (3 -1 3 (1 + θ 1 ))(1 + θ 2 ) .

(e) The final steps combine together the left and right LU factors to finally return L = ( 1 0 1 (1 + θ 1 ) 1 ) ,

(f) and U = ( 3 1 0 (3 -1 3 (1 + θ 1 ))(1 + θ 2 ) ) .

Then, in exact arithmetic, LU = ( 3 1 1 + θ 1 3(1 + θ 2 ) + θ 2 (1 + θ 1 ) 1 3 ) , which is not symmetric.

D Regularization with diagonal disorder

Wenowarrive to the more interesting part of the analysis. In Section 2 we argued that if we know the gap and the midpoint then the sign function can yield the desired spectral projector. To compute the gap and midpoint efficiently, we will take advantage of the symmetry induced by the Cholesky-based reduction, and use the Wegner estimate [138] for Hermitian diagonal perturbations to regularize the problem in the spirit of smoothed analysis. We use the following variant of the Wegner estimate.

Proposition D.1. Let G be a random diagonal matrix with independent diagonal elements sampled from N (0 , σ 2 ) , for σ = 1 2 √ 2 ln(4 n/δ ) and some δ ∈ (0 , 1 / 2) . Then for any interval I ⊂ R

where H is a fixed Hermitian matrix and I ⊂ R is a fixed interval.

Proof. This directly comes from the well-known result of Wegner [138], which states that that for any interval I ⊂ R

where ∥ ρ ∥ ∞ = 1 √ 2 πσ 2 is the supremum the probability density.

Similar results can be obtained for other classes of random matrices G . Such an example is the Gaussian Unitary Ensemble (GUE) [4], in which case G is dense, but it is invariant under rotations which might be useful for other applications.

D.1 Sampling Gaussians and perturbing in floating point

The next step is to describe how to use such random diagonal perturbations in floating point. Briefly, we first assume that we have a fixed number of bits for the floating point exponent and for the mantissa. If we sample a standard normal Gaussian (in infinite precision), it can happen that it is too large to fit in the given number of bits. This is accounted for in the failure probability, which is very small due to the decay of the normal distribution. We describe our sampling method in the following definition, which can be thought of as sampling only the most significant bits of the floating point representation of a Gaussian.

Definition D.1. A floating point standard normal sampler N ( p, t ) takes as input the number of exponent bits p , and the number of mantissa bits t , and returns a floating point number ˜ g ← fl ( g ) where g is sampled from N (0 , 1) . Following the definitions of the floating point model in Appendix A.2, if | g | ∈ [2 -M , 2 M (2 -2 -t )] , where M = 2 p -1 , then | ˜ g -g | ≤ 2 -t | g | = u | g | . If | g | > 2 M (2 -2 -t ) the sampler returns ˜ g = ± INF , i.e. the floating point representation of infinity, and if | g | < 2 -M it returns zero.

Lemma D.1 (Diagonal Gaussian sampler) . Let γ ∈ (0 , 1 / 2) and σ > 0 , and assume that we want to obtain a floating representation of the matrix γ G , where G is a diagonal matrix with independent diagonal entries from N (0 , σ 2 ) . Let N ( p, t ) be a floating point standard normal sampler as in Definition D.1, and u = 2 -t . Let ˜ σ be such that | ˜ σ -σ | ≤ η c σ , where | η c | ≤ c u 1 -c u for some (integer) constant c > 1 , and

where ˜ g i = N ( p, t ) . If p ≥ C log(log( n δ )) for some global constant C , then the diagonal matrix ˜ V = fl ( γ · ˜ σ · ˜ G ) = γ G + E , has all the following properties with probability at least 1 -δ :

(i) ˜ V i,i / ∈ { 0 , INF } ,

(ii) E is diagonal and ∥ E ∥ ≤ γη c +2 σ √ 2 ln(4 n/δ ) ,

(iii) ∥ ˜ V ∥ ≤ γ (1 + η c +2 ) σ √ 2 ln(4 n/δ ) .

Proof. Let us first analyze the conditions that are necessary such that all the sampled numbers lie within the floating point bounds from Appendix A.2, specifically, | g i | ∈ [2 -M , 2 M (2 -2 -t )] , i ∈ [ n ] , where M = 2 p -1 . For simplicity we consider | g i | ∈ [2 -M , 2 M ] .

For the lower bound, we have that

where we naively upper bounded the standard normal probability density in the interval [0 , 2 -M ] by the constant function 1 √ 2 π . For M = log 2 ( 2 n δ √ 2 π ) this implies that there are no subnormal g i for all i ∈ [ n ] simultaneously. This value of M translates to p = O ( log(log( n δ )) ) bits.

For the upper bound, a standard normal tail bound gives

For the aforementioned value of M , the probability that each one of the g i is larger than 2 M is exponentially small in δ/n , therefore we can conclude that the sampler does not return any INF values with exponentially high probability.

This is already enough to argue that the sampler returns floating point numbers that are not subnormal and not INF . However, we will need a tighter bound for the magnitude of the g i to bound the norm of the diagonal random matrix. Analyzing Eq (18) for M = 1 2 log 2 (2 ln( 4 n δ )) , and taking a union bound over all i ∈ [ n ] we have that | g i | ≤ √ 2 ln( 4 n δ ) ≪ 2 M holds for all i simultaneously with probability at least 1 -δ/ 2 .

Conditioning on the event that all the g i are in the correct range, from Definition D.1, | ˜ g i -g i | ≤ u | g i | holds for all i . Then

where c > 1 is a constant and | η c +1 | ≤ ( c +1) u 1 -( c +1) u . This means that every diagonal element of fl ( ˜ σ ˜ G ) is (1 + η c +1 ) -far from a random variable that is sampled from N (0 , σ 2 ) . Multiplying each diagonal element of fl ( ˜ σ ˜ G ) with γ to form ˜ V (similar to the Step 4 of Algorithm 5) simply increases the (relative) error to (1 + η c +2 ) . Then we can write

where V is diagonal with independent diagonal entries from N (0 , σ 2 ) and E is a diagonal matrix with diagonal elements E i,i bounded in magnitude by

It also holds that ∥ V ∥ ≤ σ √ 2 ln(4 n/δ ) since the diagonal elements V i,i since they are just the g i 's scaled by σ , and we already bounded | g i | ≤ √ 2 ln(4 n/δ ) , which implies that ∥ E ∥ ≤ γ ( c +2) u 1 -( c +2) u σ √ 2 ln(4 n/δ ) . Finally

We conditioned only on two random events, where each holds with probability at least 1 -δ/ 2 , giving the final success probability of at least 1 -δ .

REGULARIZE

Input: Hermitian matrix A ∈ H n , perturbation scale factor γ ∈ (0 , 1 / 4) , failure probability parameter δ ∈ (0 , 1 / 4) .

Requires: ∥ A ∥ ≤ 1

.

Output: Hermitian perturbed matrix ˜ X .

Ensures:

See Proposition D.2.

We can now use the Wegner estimate to get a minimum singular value bound for diagonal shifts over a grid, defined as follows.

Definition D.2 (Grid) . A 1 -d grid in the real line is a set of s points defined as

The cardinality of a grid g = grid( l, r, h ) is denoted as | g | .

The following Proposition 3.2 summarizes the properties of Algorithm 5, REGULARIZE , that we will use to perform efficient eigenvalue counting queries to compute the spectral gap.

Proposition D.2. Let A with ∥ A ∥ ≤ 1 be a Hermitian matrix, γ, δ ∈ (0 , 1 / 4) two given parameters, and ˜ X ← REGULARIZE ( A , γ, δ ) . Let g be an arbitrary (but fixed) grid of points in [ -2 , 2] with size | g | = T . For every element h i ∈ g consider the matrices M i = h i -˜ X and ˜ M = h i -˜ X + E i , where E i denote the diagonal floating point error matrices induced by the shift. If the machine precision u satisfies

which translates to O (log( Tn γδ )) bits, then all the following hold simultaneously with probability 1 -2 δ :

Proof. Let X = A + γσ G , where σ G is a diagonal matrix with independent elements from N (0 , σ 2 ) where σ = 1 2 √ 2 ln(4 n/δ ) . Let B = X /γ = A /γ + σ G . Denote by E G the event ∥ σ G ∥ ≤ 1 2 . From a tail bound as in Lemma D.1, Pr[ E G ] ≥ 1 -δ . Conditioning on E G , we obtain ∥ B ∥ = ∥ A /γ + σ G ∥ ≤ 1 /γ + 1 / 2 := β . This directly implies that Λ( B ) ⊆ ( -1 /γ -1 / 2 , 1 /γ +1 / 2) . Let h ′ i = h i /γ for all i = 1 , . . . , T .

From the Wegner estimate (Proposition D.1), for every fixed interval I ,

Setting I i to be an interval centered at h ′ i with size | I i | = δ Tn √ 4 π ln(4 n/δ ) , we have that the probability that I i contains any eigenvalues is at most δ/T . Since there are no eigenvalues in I i , this means that the smallest singular value of h ′ i -B is at least half the size of I i , i.e. σ min ( h ′ i -B ) ≥ | I i | / 2 = δ 2 Tn √ 4 π ln(4 n/δ ) . Hence, σ min ( h i -X ) = γσ min ( h ′ i -B ) ≥ γδ 2 Tn √ 4 π ln(4 n/δ ) holds as well.

From Lemma D.1, if the diagonal Gaussian sampler is called with p = Θ(log(log( n δ )) and t = log(1 / u ) then

∥

E

γη

1

c

+2

2

,

where | η c +2 | ≤ ( c +2) u 1 -( c +2) u . The elements of the diagonal error matrix E (+) in Algorithm 5 are bounded by

Then

If u ≤ ϵ 2( c +2) for some ϵ ∈ (0 , 1) then | η c +2 | ≤ ϵ and since γ < 1 / 2 we also have that ∥ ˜ X -X ∥ ≤ ϵ . Recall that ∥ X ∥ = ∥ A + γσ G ∥ ≤ 1 + γ/ 2 ≤ 5 / 4 and thus

assuming that c ≥ 1 . We now collect all the error bounds and apply Weyl's inequality to argue that

The same holds for the singular values σ j since the matrices are Hermitian. Thus if

then which implies

The aforementioned bound on ϵ also gives two (loose) bounds that are useful for the analysis of algorithms later:

and also from Weyl's inequality

The requirement for the machine precision is

bits of precision (recall that we also required p = Θ(log(log( n/δ ))) for the exponent of the Gaussian sampler, but this does not affect the total number of bits).

For the complexity, we have n calls to N ( p, t ) and O ( n ) scalar additions and multiplications thereafter, which accumulate to O ( n ) total floating point operations using O ( log( Tn γδ ) ) bits.

For the success probability, there are two types random events that we took into account: E G , which fails with probability at most δ , and the Wegner estimate for T points h i in the grid g , where each one fails with probability at most δ/T . Aunion bound over all random events gives a success probability of at least 1 -δ -Tδ/T = 1 -2 δ .

E Fast spectral gaps and eigenvalues with counting queries

The core of our methods is an algorithm to efficiently approximate spectral gaps using only 'eigenvalue counting queries,' avoiding an explicit (and expensive) diagonalization.

E.1 Counting eigenvalues

The main subroutine of our algorithm is used to count the eigenvalues that are smaller than a given threshold efficiently.

Lemma E.1. Let ˜ X be Hermitian, ∥ ˜ X ∥ ≤ 2 , and h ∈ [ -2 , 2] is a fixed point that can be exactly represented in floating point, i.e. fl ( h ) = h . There exists an algorithm COUNT ( ˜ X , h, ε ) which takes as input h, ˜ X , and ε ∈ (0 , 4 / 199) with the guarantee that σ min ( h -˜ X ) ≥ ε , and it returns an integer n ( -) , which is the precise number of eigenvalues of ˜ X that are smaller than h . The algorithm requires

which translates to arithmetic operations using

bits of precision.

Proof. Let M = h -˜ X and ˜ M = h -˜ X + E be its floating point counterpart. As the shift only distorts the diagonal elements, ∥ E ∥ = max i | E i,i | ≤ u max i | h -˜ X i,i | ≤ 4 u . By assumption σ min ( M ) ≥ ε , hence if u ≤ ε/ 8 then Weyl's inequality implies σ min ( M + E ) ≥ σ min ( M ) -ε/ 2 ≥ ε/ 2 .

Assume that we want to approximate n ( -) by the expression

for some α SGN , η SGN , ϵ SGN (to be determined). For simplicity we will use SGN ( ˜ M ) to denote SGN ( ˜ M , α SGN , η SGN , ϵ SGN ) . Assuming for now that SGN converged to the requested error ϵ SGN , we know that ∥ sgn( ˜ M ) -SGN ( ˜ M ) ∥ ≤ ϵ SGN . To proceed recall that for any matrix A

which can be achieved with a binary tree-type addition.

Now consider the error

It remains to bound ∥ sgn( M ) -sgn( ˜ M ) ∥ . For this we can directly use Lemma B.1 where we set S = I , H = ˜ X , and µ = h . The lemma states that if ∥ E ∥ ≤ ϵ shift | λ min ( h -˜ X ) | 2 π 128 then ∥ sgn( h -˜ X ) -sgn( h -˜ X + E ) ∥ ≤ ϵ shift for some ϵ shift ∈ (0 , 1) that we can choose. By assumption | λ min ( h -˜ X ) | ≥ ε , which means that

For SGN , we set η SGN = ε 4 , α SGN = 4 -ε 4+ ε and ϵ SGN = 1 8 n . Note that α SGN satisfies the requirement 99 / 100 < α SGN < 1 as long as ε < 4 / 199 ). Then based on Theorem A.4 SGN requires

iterations and

bits of precision.

If we set ϵ shift = 1 / 8 , then the bound for u becomes

which means that if we round to the closest integer then round( ˜ n ( -) ) = n ( -) . Note that log(1 / u ) = O ( log( n ϵ ) ) which is dominated by the bit requirements of SGN .

is sufficient to guarantee that in which case

Then (19) becomes

E.2 Approximate midpoint and gap

Having an efficient way to count eigenvalues smaller than a threshold, we show how to approximate spectral gaps. We can now prove Proposition 3.1, which we restate below for readability.

Proposition E.1. Problem 3.1 can be solved in time O ( log( 1 ϵ gap k ) q ( 1 ϵ gap k ) ) .

Proof. We start with γ 0 = 1 / 4 , and make a grid g = grid( -1 , 1+ γ, γ ) which consists of the points {-1 , -3 / 4 , -2 / 4 , -1 / 4 , 0 , 1 / 4 , 2 / 4 , 3 / 4 , 1 , 5 / 4 } (a useful note for later is that all the numbers in the grid can be exactly represented in floating point). Let count( h j ) denote the number of γ 0 -distorted values λ ′ i that are smaller than h j , for some h j ∈ g , j = 1 , . . . , 10 . Query all h j ∈ g for count( h j ) . Set j k = arg min j { h j | count( h j ) ≥ k } and j k +1 = arg min j { h j | count( h j ) ≥ k +1 } . Now λ ′ k ∈ [ h j k -1 , h j k ] and λ ′ k +1 ∈ [ h j k +1 -1 , h j k +1 ] . The points in the grid are equispaced so each interval has length γ . Since by definition λ ′ k ∈ [ λ k -γ 0 , λ k + γ 0 ] , then λ k ∈ [ h j k -1 -γ 0 , h j k + γ 0 ] = [ h j k -2 , h j k +1 ] . Similarly for λ k +1 . We have now restricted both λ k and λ k +1 inside some intervals of length 3 γ 0 . Let us denote those intervals as I k = [ h j k -2 , h j k +1 ] and I k +1 = [ h j k +1 -2 , h j k +1 +1 ] . In the next step we halve γ 0 to γ 1 = 1 / 8 . We now create two grids, one inside I k and one inside I k +1 : g k = grid( h j k -2 , h j k +1 + γ 1 , γ 1 ) and similar for g k +1 . Each of the new grids has exactly 8 points and each point can be exactly represented in floating point. As in the previous iteration, we query all the 8 points h j of g k for count( h j ) , and we pick a new j k = arg min { h j | count( h j ) ≥ k } , and we do the same for j k +1 . There are a total of 16 queries. As before, we have now restricted λ k and λ k +1 inside two new intervals I k and I k +1 where | I k | = | I k +1 | = 3 γ 1 , which is half the size of the corresponding intervals of the previous iterations. If we set ˜ λ k equal to the midpoint of I k , then | ˜ λ k -λ k | ≤ 3 γ/ 2 . The same for λ k +1 . We keep repeating the same procedure by halving γ i in each step i .

It remains to find a proper termination condition. Let us denote ˜ µ k = ˜ λ k + ˜ λ k 2 and ˜ gap k = ˜ λ k -˜ λ k +1 , where ˜ λ k and ˜ λ k +1 are as above. Then, after m steps,

Thus, γ m ≈ ϵ gap k / 3 is a sufficient terminating criterion. We don't know gap k , but we can use ˜ gap k instead, i.e., γ ≤ ϵ ˜ gap k -3 γ m 3 is also a sufficient, which gives a quantifiable termination criterion γ m ≤ ϵ ˜ gap k / 6 . But since ˜ gap k ≥ gap k -3 γ m , then the terminating criterion will be reached in the worst case when

meaning that the algorithm will halt and return an ϵ -accurate ˜ µ k , ˜ gap k in at most O ( log( 9 γ m ) ) =

Each h j -query at iteration i costs O (polylog(1 /γ i )) by assumption, and there are exactly 16 h j -queries in each iteration, which gives a total query cost of O (polylog(1 /γ i )) per iteration, which is maximized for the smallest γ m = Θ( ϵ gap k ) .

We can now describe the algorithm GAP that computes the k -th gap and the midpoint of a Hermitian definite pencil.

Theorem E.1 ( GAP , Restatement of Theorem 3.1) . Let H ∈ H n , S ∈ H n ++ and ∥ H ∥ , ∥ S -1 ∥ ≤ 1 , which define a Hermitian definite pencil ( H , S ) . Given k ∈ [ n -1] , accuracy ϵ ∈ (0 , 1) , and failure probability δ ∈ (0 , 1 / 2) , there exists an algorithm

which returns ˜ µ k = µ k ± ϵ gap k and ˜ gap k = (1 ± ϵ ) gap k , where µ k = λ k + λ k +1 2 and gap k = λ k -λ k +1 . The algorithm requires

arithmetic operations using O ( log( n ) ( log 4 ( n δϵ gap k ) + log( κ ( S )) )) bits, where λ i are the eigenvalues of ( H , S ) . If κ ( S ) is unknown, additional O ( T MM ( n ) log( nκ ( S ) δ ) log( κ ( S ))) floating point operations and O (log( n ) log 4 ( nκ ( S ) δ )) bits are suficient to compute it with Corollary E.1.

Proof. We first set γ 0 = 1 / 8 and call ˜ H = REDUCE ( H , S , γ 0 4 ) . From Proposition C.3, REDUCE requires O ( T MM ( n )) floating point operations using O (log( n ) log( κ ( S )) + log(1 /γ 0 )) bits, and it returns a matrix ˜ H that satisfies | λ i ( ˜ H ) -λ i ( H , S ) | ≤ γ 0 4 .

Wethen call ˜ X ← REGULARIZE ( ˜ H , γ 0 2 , δ 0 ) , where δ 0 = δ/ 2 is the initial failure probability. From Proposition 3.2, for all i it holds that | λ i ( ˜ X ) -λ i ( ˜ H ) | ≤ 9 γ 0 / 16 . Therefore, all the eigenvalues, which initially lie in [ -1 , 1] , are distorted by at most γ 0 .

Next, recall the counting query model of Proposition 3.1: in the first step we construct a grid g = grid( -1 , 1 + γ 0 , γ 0 ) , where | g | = 10 . From Proposition 3.2, the regularization ensures that for every h j ∈ g it holds that σ min ( h j -˜ X + E ) ≥ ε 0 for ε 0 = γ 0 δ 0 8 | g | n √ π ln(4 n/δ 0 ) (this is included in the success probability of 1 -δ 0 of Proposition 3.2). Then we execute COUNT ( ˜ X , h j , ε 0 ) for every h j . We continue by halving at each step both γ and δ , constructing the corresponding grids as per Proposition 3.1, and counting eigenvalues over the grid. In each iteration after the first one, we keep track of two grids g with size | g | = Θ(1) each, and therefore we can ignore | g | in the complexity.

After a total of m = O ( log( 1 ϵ gap k ) ) iterations, we have that γ m = Θ( ϵ gap k ) and δ m = Θ( δ 0 ϵ gap k ) . Invoking Lemma E.1, and plugging in the value of ε m , each call to COUNT during the entire algorithm costs at most

arithmetic operations using O ( log( n ) log 3 ( 1 ε m ) log( n ε m ) ) = O ( log( n ) log 4 ( n γ m δ m ) ) bits. The last iteration where γ m = Θ( ϵ gap k ) and δ m = Θ( δ 0 ϵ gap k ) is the most expensive, which gives a total of

arithmetic operations and

bits. The failure probability is δ 0 in the first iteration and halved at each subsequent iteration, in which case a union bound converges to 2 δ 0 = δ .

To be able to call REDUCE in every step, we need to approximate κ ( S ) in order to set the machine precision appropriately. For this we can use Corollary E.1, which is detailed in the next section, and it returns ˜ κ ∈ Θ( κ ( S )) with probability 1 -δ . It requires O ( T MM ( n ) log( nκ ( S ) δ ) log( κ ( S ))) floating point operations using O (log( n ) log 4 ( nκ ( S ) δ )) bits.

E.3 Singular values, singular gaps, and condition number

The gap finder can be extended to compute eigenvalues and singular values and singular gaps or arbitrary matrices instead of eigenvalue gaps of Hermitian matrices. The following algorithm in Proposition E.2 computes the k -th singular value of a matrix, for arbitrary k ∈ [ n ] , using counting queries.

Proposition E.2 ( SIGMAK ) . Given a matrix A ∈ C m × n , n ≥ m with ∥ A ∥ ≤ 1 , an integer k ∈ [ n ] such that rank( A ) ≥ k , an accuracy ϵ ∈ (0 , 1) , and failure probability δ ∈ (0 , 1) , there exists an algorithm

which returns a value ˜ σ k ∈ (1 ± ϵ ) σ k ( A ) with probability at least 1 -δ . The algorithm executes

floating point operations and requires O ( log( n ) log 4 ( n δϵσ k ) ) bits.

Notes: If m > n then we can apply the algorithm on A ⊤ . If ∥ A ∥ > 1 , we can use Theorem A.7 to approximate ∥ A ∥ and scale accordingly. The rank( A ) ≥ k assumption can potentially be omitted by a more sophisticated counting query strategy.

Proof. We will solve the problem with counting queries, similar to Theorem 3.1, halving γ and δ in every iteration. We start with γ 0 = 1 / 8 and δ 0 = δ/ 2 . First, we divide A by two, to ensure that ∥ A ∥ ≤ 1 / 2 . Then we construct ˜ A = HERM ( MM ( A , A ∗ )) = AA ∗ / 4 + E MM . To perform this rectangular multiplication in practice, we partition A in m/n blocks of size n × n each, perform the individual multiplications and sum the results. We can ensure that ∥ E MM ∥ ≤ γ 0 / 4 if we use O (log( n γ 0 )) bits. Then from Weyl's inequality | λ k ( ˜ A ) -λ k ( AA ∗ / 4) | ≤ γ 0 / 4 . We then call ˜ X ← REGULARIZE ( ˜ A , γ 0 , δ 0 ) . Conditioning on success of Proposition 3.2, for all i it holds that | λ i ( ˜ X ) -λ i ( ˜ A ) | ≤ 9 γ 0 / 16 , and therefore | λ i ( ˜ X ) -λ i ( AA ∗ / 4) | ≤ γ 0 .

Let λ ′ i = λ i ( ˜ X ) and λ i = λ i ( AA ∗ / 4) . Since all the λ i are non-negative, then all the λ ′ i ∈ [ -γ 0 , 1 + γ 0 ] . We thus construct grid g = grid(0 , 1 + γ 0 , γ 0 ) = { 0 , 1 / 8 , 2 / 8 , 3 / 8 , . . . , 1 , 9 / 8 } . We now need to perform counting queries similar to Theorem 3.1. Since we conditioned on success of Proposition 3.2, REGULARIZE ensures that for every h j ∈ g it holds that σ min ( h j -˜ X + E h j ) ≥ ε 0 for ε 0 = γ 0 δ 0 8 | g | n √ π ln(4 n/δ 0 ) . Then we execute C j ← COUNT ( ˜ X , h j , ε 0 ) for every h j ∈ g (in total Θ(1) calls to COUNT ). We set j k to be the smallest j such that C j ≥ k . Now λ ′ 1 ∈ [ h j k , h j k +1 ] . Since λ ′ k ∈ [ λ k -γ 0 , λ k + γ 0 ] , then λ k ∈ [ h j 1 -1 -γ 0 , h j k + γ 0 ] = [ h j k -2 , h j k +1 ] := I k , where | I k | = 3 γ 0 (we can always ignore the negative portion of I k , if there is any).

In the next step we halve γ 0 to γ 1 = 1 / 8 , and we create another grid inside I k , g 1 = grid( h j k -2 , h j k +1 + γ 1 , γ 1 ) . The new grid has at most 8 points. We query all the points h j of g 1 for C j ← COUNT ( ˜ X , h j , ϵ 1 ) , and we pick a new j 1 = arg min { h j | C j ≥ k } . Now λ k lies inside a new interval I k with | I k | = 3 γ 1 , which is half the size of the interval of the previous iteration. If we set ˜ λ k equal to the midpoint of I k (which is always positive), then | ˜ λ k -λ k | ≤ 3 γ/ 2 . We keep repeating the same procedure by halving γ i and δ i in each step i .

After m steps we have that

Asufficient termination criterion is γ m ≈ ϵλ k . Instead of λ k (which is unknown) we use ˜ λ k instead. Then γ ≤ ϵ ˜ λ k -3 γ m 3 is also a sufficient, which gives a termination criterion γ m ≤ ϵ ˜ λ k / 6 that we can actually calculate. In turn, this termination criterion will be reached in the worst case when

meaning that the algorithm will halt and return an ϵ -accurate λ k in at most m = O ( log( 9 γ m ) ) = O ( log( 1 ϵλ k ) ) iterations.

Note that in each iteration we make a total of Θ(1) calls to COUNT . From Lemma E.1, if we plug in the corresponding ε i , each call to COUNT during the entire algorithm costs at most

arithmetic operations using O ( log( n ) log 4 ( n γ i δ i ) ) bits. The cost is maximized in the last iteration where γ m = Θ( ϵλ k ) and δ m = Θ( δ 0 ϵλ k ) , which gives

arithmetic operations and

bits. As in Theorem 3.1, since the failure probability is halved in each iteration, a union bound converges to 2 δ 0 = δ .

Now we have that | λ k -˜ λ k | ≤ ϵλ k . Recall that σ k ( A ) = √ λ k ( AA ∗ ) = 2 √ λ k , since we computed AA ∗ and divided by two. If we set σ k = 2 √ ˜ λ k then

˜

where the last holds since √ 1 + ϵ ≤ 1 + ϵ and 1 -ϵ ≤ √ 1 -ϵ for ϵ ∈ (0 , 1) . Note that we lazily assumed that we computed ˜ σ k = 2 √ ˜ λ k exactly, which does not hold, the square root introduces a machine precision error. The calibration is left as an exercise. If we replace λ k with σ 2 k / 4 and δ 0 = δ/ 2 in the complexity bounds we get the desired

arithmetic operations and O ( log( n ) log 4 ( n δϵσ k ) ) bits.

We now have a tool for the condition number.

Corollary E.1 ( COND ) . Let A ∈ H n ++ . Given δ ∈ (0 , 1 / 2) , we can compute ˜ κ such that κ ( A ) ≤ ˜ κ ≤ 32 κ ( A ) , with an algorithm ˜ κ ← COND ( A , δ ) in

using O ( log( n ) log 4 ( nκ ( A ) δ ) ) bits of precision with probability at least 1 -δ .

Proof. We first compute ˜ Σ ∈ [0 . 9 ∥ A ∥ , 2 ∥ A ∥ ] using Corollary A.1 with parameter δ/ 3 in O ( n 2 log( n ) log(1 /δ )) floating point operations using O (log( n )) bits of precision. The algorithm succeeds with probability at least 1 -2 δ 3 . We then scale A ′ ← A /M where M is the smallest power of two that is larger than 4 ˜ Σ . This implies 1 16 ≤ ∥ A ′ ∥ ≤ 1 3 . 6 , and also σ min ( A ′ ) ∈ [ σ min ( A ) 16 ∥ A ∥ , σ min ( A ) 3 . 6 ∥ A ∥ ] = [ 1 16 κ ( A ) , 1 3 . 6 κ ( A ) ] . Thus, it suffices to approximate σ min ( A ′ ) and scale it to obtain the desired approximation for κ ( A ) .

We now call ˜ σ ′ min ← SIGMAK ( A ′ , 1 , 1 / 2 , δ/ 3) , which succeeds probability 1 -δ/ 3 and returns ˜ σ ′ min ∈ (1 ± 1 2 ) σ min ( A ′ ) . It requires

arithmetic operations and

bits. Then we can set ˜ κ = 1 ˜ σ ′ min ∈ [ κ ( A ) , 32 κ ( A )] . The success probability is 1 -2 δ 3 -δ 3 = 1 -δ .

E.4 Proof of Theorem 1.1

We now have all the prerequisites to prove that Algorithm 1 provides the guarantees of our main Theorem 1.1, which we restate for readability.

Theorem E.2 (Restatement of Theorem 1.1) . Let ( H , S ) be a Hermitian definite pencil of size n , with ∥ H ∥ , ∥ S -1 ∥ ≤ 1 , and λ 1 ≤ λ 2 ≤ . . . ≤ λ n its eigenvalues. There exists an algorithm

which takes as input H , S , an integer 1 ≤ k ≤ n -1 , an error parameter ϵ ∈ (0 , 1) and returns a matrix ˜ Π k such that

where Π k is the true spectral projector on the invariant subspace that is associated with the k smallest eigenvalues. The algorithm executes

floating point operations, using

bits of precision, where κ ( S ) = ∥ S ∥∥ S -1 ∥ and gap k = λ k +1 -λ k . Internally, the algorithm needs to generate a total of at most ˜ O ( n ) standard normal floating point numbers using additional O (log(log( n ))) bits. In the regular case, when S = I , the O (log( nκ ( S )) log( κ ( S ))) term in the arithmetic complexity and the O (log 4 ( nκ ( S ))) term in the number of bits are removed.

Proof. We first compute ˜ µ k , ˜ gap k , ˜ κ which satisfy the requirements of Proposition 2.1 using COND and GAP . Afterwards we can use them to call PURIFY to obtain the spectral projector ˜ Π k such that ∥ ˜ Π k -Π k ∥ ≤ ϵ . We first call COND ( S , 1 4 , 1 2 n ) , which, from Corollary E.1, requires

bits. Then, given the result of COND , we call GAP ( H , S , k, 1 8 , 1 2 n ) which requires

bits. They both succeed at the same time with 1 -1 /n probability. PURIFY ( H , S , ˜ µ k , ˜ gap k , ˜ κ, ϵ ) requires

floating point operations and arithmetic operations and

arithmetic operations and

bits. The total arithmetic complexity is therefore

and the bit requirement is

Notice that the ϵ term appears only inside log log in the arithmetic complexity and in only one of the log factors of the number of bits.

F DFT background

Density Functional Theory (DFT) [88], which was awarded the Nobel prize in Chemistry in 1998, is considered as one of the most common methods to perform electronic structure calculations thanks to its ab initio character. That means that no input besides the initial atomic structure is required to determine the relaxed geometry, energy levels, and the electron structure of a given material, in their bulk or nanostructured form. The main idea behind DFT is to describe the system and its properties by the electron density only. DFT calculations are widely used in industry and in academia to predict, for example, the properties of novel materials [102] or to optimize the performance of batteries [69], solar cells [142], or nanoelectronic devices [85]. Many scientific libraries implementing DFT algorithms have been developed, which persistently occupy supercomputing clusters and receive up to tens of thousands of citations annually at the time of this writing [89, 59, 127, 74]. Despite the success of DFT calculations, important theoretical aspects of their underlying algorithms remain unclear.

Assume a system that is composed of a set of n a ∈ N atoms positioned inside a fixed threedimensional domain. The Kohn-Sham equation in real-space describes the electronic wave function ψ ( r ) of the system

where r = ( x, y, z ) is any position in the domain. Here H ( r ) is the so-called single-particle Hamiltonian operator of the system, which can be written in the following form:

where the first term is the kinetic term, the second term accounts for electron-ion interactions, the third term represents the electron-electron interactions, which are solved at the Hartree level through Poisson's equation, and the fourth term is the so-called exchange-correlation term. The electron density n ( r ) is the main observable of interest in DFT. It is related to the solutions E i and ψ i ( r ) of Eq. (20) through

where E F is the Fermi level of the system, E i is the energy level (eigenvalue) corresponding to the wave function ψ i ( r ) , and f ( E i ; E F ) is the occupation term. At zero temperature ( T = 0 K ) , f ( E i ; E F ) is equal to one for occupied states ( E i < E F ) and zero otherwise. 3 The E F , E i , and ψ i are the unknown quantities for which there exists no analytical solution, except for some special cases. Approximate numerical solutions can be sought after expanding the wave functions into a suitable basis. In this case, the ψ i 's are approximated by a set of n = Θ( n a ) basis functions χ j ( r ) , j ∈ [ n ] , typically localized around atomic positions. 4 Common basis sets that lead to efficient algorithms to solve the eigenvalue problem of Eq. (20) include Linear Combination of Atomic Orbitals (LCAO) [32], Gaussian-Type Orbitals (GTO) [22], or Maximally Localized Wannier Functions (MLWF) [101]. In this context, the wave functions take the form of linear combinations of the n localized orbitals. In particular, for all i ∈ [ n ]

where c ij are the (unknown) complex coefficients. This expansion can also be written in matrix notation as

where y ( r ) ∈ C n and x ( r ) ∈ C n are (complex) vectors and C is a n × n matrix. By expanding the wave functions into LCAO, GTO, or MLWF basis, the Hamiltonian operator becomes a matrix

Similarly, an overlap matrix S must be introduced

which is equal to the identity matrix I if the basis elements are orthogonal (e.g. S = I for MLWF). The entries of H and S represent the interaction between two basis elements that can be located on the same or on different atoms. Furthermore, due to the complex conjugate property of the inner product, both H and S are Hermitian matrices. Moreover, S is positive-definite ([103, Chapter 3]). Putting everything together, the expansion coefficients c ij are obtained by solving the following generalized eigenvalue problem as already formulated in Eq. (1):

The so-called density matrix P can be derived from the C and Λ solutions

The density matrix possesses interesting properties. The matrix PS is a projector matrix and therefore it is idempotent, i.e. PS = ( PS ) 2 . Moreover, it can be used to derive the electron density at any position r

where x ( r ) ∈ C n is a vector such that x i ( r ) = χ i ( r ) , i.e. x ( r ) contains the values of all the basis functions at position r . Given these basic definitions, we next give an overview of existing algorithms.

F.1 Linear scaling methods

Because of the strong localization of the χ i ( r ) 's, the ( i, j ) interactions rapidly decay with the distance between the atoms and can be discarded beyond a pre-defined cut-off radius. As a consequence, H and S are often banded matrices with sometimes a sparse band. It should also be noted that at equilibrium, the number of negatively charged electrons present in the system (which corresponds to the integral of n ( r ) over space) must exactly compensate the number of positively charged protons Z p . This allows for the computation of the Fermi level: from Eq. (25) and the definition of S , it follows that trace( PS ) = Z p , which means that one can perform a binary search on the Fermi level E F until trace( PS ) = Z p is satisfied. These observation has led to the development of so-called linear scaling methods, which aim to exploit the sparsity of the matrices to obtain faster solutions [143, 57, 86, 87, 56, 62, 63]. Typically, the density matrix and the Fermi level are iteratively computed, until all required constraints are satisfied, i.e., the density matrix should be idempotent and trace( PS ) = Z p . This can be done, for example, by approximating the matrix sign function using a Newton iteration [136]. To optimize performance, heuristics such as truncating matrix elements with negligible magnitude are applied, at the cost of decreasing the solution accuracy. Empirical evidence shows that such methods tend to scale nearly linearly to the system size, however, they do not exhibit provable theoretical guarantees. Similar methods have been studied based on finite-differences, which aim to approximate the Fermi distribution with polynomial expansions [145, 144]. Finally, the work of [65, 137] offer the possibility to achieve lower complexities than O ( n 3 ) for both the eigenvalues and eigenvectors of structured matrices, by leveraging the fast multipole method. However, an end-to-end stability analysis of those algorithms remains open. Indeed, we are not aware of any end-to-end analysis with provable approximation guarantees that has lower than O ( n 3 ) worst-case complexity for any of the aforementioned algorithms.

F.2 Density matrix

We can directly apply Theorem 1.1 to compute density matrices in DFT.

The following Theorem summarizes the analysis of Algorithm 6 for the density matrix.

˜

DENSITY

Input:

Hermitian definite pencil H ∈ H n , S ∈ H n ++ , number of occupied orbitals k .

Requires: ∥

H ∥ ≤ 1 , ∥ S -1 ∥ ≤ 1 , k ∈ [ n -1] .

1: ˜ Π ← PROJECTOR ( H , S , k, ϵ 32 ) .

2: S INV ← INV ( S ) .

3: ˜ P ← HERM ( MM ( ˜ Π , MM ( S INV , ˜ Π ∗ )) ) .

4: return ˜ P .

Output: Approximate density matrix P .

Ensures: ∥ ˜ P -P ∥ ≤ ϵ with probability at least 1 -O (1 /n ) .

Algorithm 6: DENSITY .

Theorem F.1 ( DENSITY ) . Given a Hermitian definite pencil ( H , S ) with ∥ H ∥ , ∥ S ∥ ≤ 1 , an integer k ∈ [ n -1] denoting the number of occupied states in the system, and ϵ ∈ (0 , 1) , Algorithm 6 returns a matrix ˜ P ← DENSITY ( H , S , k, ϵ ) such that

where P is the true density matrix of the system and succeeds with probability at least 1 -O (1 /n ) . The floating point arithmetic complexity is

and it requires

bits of precision.

Proof. In the first step of the algorithm we compute ˜ Π such that

where Π is the spectral projector on the invariant subspace of the definite pencil associated with the k smallest eigenvalues.

Then we invert S , and the final step of the algorithm computes ˜ P . Unrolling all the computations, ˜ P can be written as ΠS -1 Π ∗ + E P where E P is a Hermitian error matrix (yet to be detailed). It is not hard to verify that ΠS -1 Π ∗ is equal to P , i.e. the true density matrix.

It thus remains to bound ∥ E P ∥ . We can directly force ∥ E INV 1 ∥ ≤ ϵ Π ∥ S -1 ∥ ≤ ϵ Π by setting u ≤ ϵ Π 1 µ INV ( n ) κ ( S ) c INV log( n ) . Then from line 5 of Algorithm 6

We bound each term as follows.

Putting everything together we have that

Note that the aforementioned value of u is already dominated by the requirement of PROJECTOR . The same holds for the arithmetic complexity. Therefore, the arithmetic complexity, the success probability, and the number of required bits, similar to those of Theorem 1.1 up to a constant factor.

F.3 Electron density queries

Having a good approximation of the density matrix, we can use it to query for the electron density at any point of interest. A direct application of Theorem F.1 gives the following.

Corollary F.1. Given a position r in the atomic domain, let x be a vector x = ( χ 1 ( r ) χ 2 ( r ) . . . χ n ( r )) , where χ i are the basis functions that were used to construct the Hamiltonian and overlap matrices H , S . Let ˜ P ← DENSITY ( H , S , ϵ, k ) where k is the number of occupied states in the atomic system, and let P be the true density matrix. Let n ( r ) = x ∗ Px be the true electron density at r and ˜ n ( r ) = x ∗ ˜ Px . Then as long as DENSITY succeeds,

Proof. First we recall that ϵ ∈ (0 , 1 / 12) and note that ∥ ˜ P ∥ ≤ ∥ P ∥ + ϵ ≤ 1 + ϵ. The quadratic form x ∗ ˜ Px is computed in two steps: y ← fl ( ˜ Px ) and ˜ n ( r ) = fl ( x ∗ y ) . From [70], Eq. (3.12) and Lemma 6.6, we know that the matrix-vector product satisfies y = ˜ Px + e where ∥ e ∥ = ∥ ˜ Px -y ∥ ≤ √ nη n ∥ ˜ P ∥∥ x ∥ , and η n := n u 1 -n u . For the chosen u we have √ nη n ≤ 2 ϵ . From the same equations, the subsequent dot product satisfies ˜ n ( r ) = fl ( x ∗ y ) = x ∗ y + ϵ where | ϵ | ≤ η n ∥ x ∥∥ y ∥ ≤ η n ∥ x ∥ ( ∥ ˜ Px ∥ + ∥ e ∥ ) ≤ η n ∥ ˜ P ∥∥ x ∥ 2 (1 + √ nη n ) ≤ 4 ϵ ∥ x ∥ 2 . Then | n ( r ) -˜ n ( r ) | = | x ∗ Px -( x ∗ y + ϵ ) | = | x ∗ Px -x ∗ ˜ Px -x ∗ e -ϵ | ≤ | x ∗ Px -x ∗ ˜ Px | + | x ∗ e | + | ϵ | ≤ 2 ϵ ∥ x ∥ 2 + 2 ϵ ∥ x ∥ 2 +4 ϵ ∥ x ∥ 2 = 8 ϵ ∥ x ∥ 2 .

This can be generalized for many different points r . If N is the number of those points and N = Θ( n ) (which is typically the case in applications), then one can stack the vectors x ( r 1 ) , x ( r 2 ) , . . . , x ( r N ) as the rows of a matrix X and use fast matrix multiplication to compute all n ( r i ) simultaneously, instead of querying each n ( r i ) one-by-one.

If N is asymptotically larger than the system size, i.e., N = O (poly( n )) , the query complexity can be reduced by applying the Johnson-Lindenstrauss (JL) lemma [78], at the cost of a polynomial dependence on the accuracy. In brief, since the matrix PS is an orthogonal projector, and therefore idempotent, it can be deduced that

where L is the lower triangular Cholesky factor of S -1 . Then ∥ x ∗ PL -∗ ∥ 2 can be approximated by ∥ x ∗ PL -∗ T ∥ 2 , where T is a n × r sparse random sign matrix scaled by 1 / √ r [1], with r = O (log( N ) /ϵ 2 ) columns. The representation of T requires O ( n × r ) = O ( n log( N ) /ϵ 2 ) random bits. Alternative random matrix distributions satisfying the JL property, with varying sparsity, runtime,

and dimension requirements exist in the literature [3, 82, 110]. The value of r is worst-case optimal [93], however, tighter bounds have been studied in the literature [21, 126, 23, 49], thus potentially leading to more practical algorithms. The full analysis is omitted.

G Deflation and bit complexity of PCA

Since its introduction in the early twentieth century [116, 72], Principal Component Analysis is one of the most important tools in statistics, data science, and machine learning. It can be used, for example, to visualize data, to reduce dimensionality, or to remove noise in measurements; cf. [79, 45] for reviews on the vast bibliography. In its simplest formulation, given a (centered) data matrix X ∈ R m × n , the goal is to find a k -dimensional embedding C k , where k < n , that maximizes the sample variance, which can be written as an optimization problem

where H = X ⊤ X ∈ R n × n is the sample covariance. It can be shown that the solution C k corresponds to the principal k singular vectors of H , i.e. the ones that correspond to the largest k singular values. Evidently, since the sample covariance is always symmetric and positive semi-definite, this can be written as a Hermitian eigenvalue problem

(which is indeed a definite GEP as in Equation (1) with S = I ). This way we can project the data in k dimensions by computing XC k , preserving as much of the variance in k dimensions as possible. Classically, PCA can be solved by approximating the SVD of H via diagonalization.

G.1 Vanilla PCA

With this short introduction, we can now describe how to enhance the analysis of PCA algorithms by using our results for spectral projectors to obtain forward-error guarantees for all the SVDrelated computations. Using our Theorem 1.1, the SIGMAK algorithm of Proposition E.2, and the DEFLATE algorithm of [19], as described in Algorithm 7, we can state the following result for the classic ('vanilla') PCA, which provides forward error guarantees in matrix multiplication time regardless how large k is. This is can serve as the backbone for more advanced algorithms that are analyzed in the next section, which typically first use random projections to reduce the matrix size to approximately O ( k × k ) and then perform the SVD computation on the smaller matrix.

Remark G.1. For simplicity, we have assumed that the covariance matrix contains no errors. This might not be true if H is computed numerically from the data matrix X . It is not hard to extend the algorithm to take this into account as well, and nothing changes in the analysis (except some negligible constant factors). To handle errors in the input matrix one can adapt the analysis from the next Section G.2.

Theorem G.1 (PCA) . Let H be an n × n symmetric sample covariance matrix of a centered data matrix X ∈ R m × n , i.e. H = X ⊤ X , ∥ H ∥ ≤ 1 , k ∈ [ n ] is a target rank, and ϵ ∈ (0 , 1) an accuracy parameter. Then we can compute a matrix ˜ C k with k columns such that ∥ X -X ˜ C k ˜ C ⊤ k ∥ ≤ (1 + ϵ ) ∥ X -XC k C ⊤ k ∥ , where C k ∈ R n × k contains the topk (right) singular vectors of X in

arithmetic operations using O ( log( n ) ( log 4 ( n ϵ gap k ) + log 4 ( n σ k +1 ) ) +log( 1 ϵσ k +1 ) ) bits of precision, with probability at least 1 -O (1 /n ) .

Proof. Wefirst compute ˜ σ k +1 ← SIGMAK ( H , k +1 , 1 2 , 1 n ) such that, from Proposition E.2, ˜ σ k +1 ∈ (1 ± 1 2 ) σ k +1 ( H ) with probability 1 -1 /n . It requires

PCA

Input: Sample Covariance Matrix H ∈ R n × n of a centered data matrix X ∈ R m × n , target rank k , accuracy ϵ ∈ (0 , 1) .

Requires: ∥ H ∥ ≤ 1 , k ∈ [ n -1] .

4: ˜ C k ← DEFLATE ( ˜ Π k , k, ( ϵ ′ 2 20 3 n 4 ) 2 , ϵ ′ ) .

5: return ˜ C k .

Output: Approximate principal component matrix ˜ C k .

Ensures: ∥ X -X ˜ C k ˜ C ⊤ k ∥ ≤ (1 + ϵ ) ∥ X -XC k C T k ∥ with probability at least 1 -O (1 /n ) .

Algorithm 7: PCA .

floating point operations and O ( log( n ) log 4 ( n σ k +1 ) ) bits. Then we set ϵ ′ ← ϵ ˜ σ k +1 / 4 . In the next step, we compute ˜ Π k ← PROJECTOR ( H , I , k, ( ϵ ′ 2 20 3 n 4 ) 2 ) , which, from Theorem 1.1, returns a spectral projector ˜ Π k that satisfies ∥ ˜ Π k -Π k ∥ ≤ ( ϵ ′ 2 20 3 n 4 ) 2 with probability 1 -1 /n . It requires

arithmetic operations and O ( log( n ) log 4 ( n ϵ ′ gap k ) ) = O ( log( n ) log 4 ( n ϵσ k +1 gap k ) ) bits. Then we use DEFLATE to compute the matrix C ′ k ← DEFLATE ( ˜ Π k , k, ( ϵ ′ 2 20 3 n 4 ) 2 , ϵ ′ ) . From Theorem A.5, DEFLATE succeeds with probability

and it requires O ( T MM ( n )) floating point operations using O (log( n/ϵ ′ )) = O (log( n/ ( σ k +1 ϵ ))) bits, and it internally generates O ( n 2 ) random complex normal variables. On success it returns C ′ k ∈ C n × k such that ∥ C k -C ′ k ∥ ≤ ϵ ′ , where C k is a matrix whose columns form an orthonormal basis for Π k . We can then keep only the real part of C ′ k , and set ˜ C k ← Re ( C ′ k ) . The spectral norm can only decrease by removing the imaginary part, therefore ∥ ˜ C k -C k ∥ ≤ ϵ ′ holds as well. Now we can write ˜ C k = C k + E where ∥ E ∥ ≤ ϵ ′ , which means that

where in the last we used the fact that

The result follows by rescaling ϵ by a constant and by summing together the individual algorithm complexities and bits.

G.2 Low-rank PCA

In many applications of PCA the parameter k is chosen to be small, i.e. k ≪ d . This has led to an extensive research area on the so-called low-rank approximation algorithms for PCA. One of the earliest works which introduced this type of randomized low-rank approximations is [53, 54]. Some other landmark works in the field include the analysis of randomized PCA and low-rank approximation [100, 67, 120, 33, 34], and the pioneering Block-Krylov PCA of [106], which is essentially optimal in the matrix-vector query model [125]. The approximation accuracy of low-rank approximation-based PCA methods is often measured with respect to the spectral or the Frobenius norm error, i.e. the matrix ˜ C k that is returned should satisfy C ⊤ k C k = I k × k and:

Most of the aforementioned low-rank approximation type algorithms assume exact arithmetic. A subtle detail is that they often rely on the computation of an (exact) SVD of some smaller submatrix, which is not realistic due to the Abel-Ruffini theorem. This can be justified for practical reasons since SVD can be approximated in polynomial time to arbitrary accuracy using classical solvers [64]. Quoting [67]:

'Techniques for computing the SVD are iterative by necessity, but they converge so fast that we can treat them as finite for practical purposes. '

However, if one wants to rigorously prove forward error approximations and end-to-end complexity upper bounds, there is necessarily a dependence on the singular value gap that separates the principal invariant subspace from the rest of the spectrum (consider a small perturbation of the 2 -by2 identity matrix and k = 1 as a straightforward example).

Assuming an exact SVD algorithm, the seminal analysis of the Block-Krylov PCA of [106] can in fact provide per-singular vector guarantees, which are much stronger than the classical norm-wise bounds. Then arithmetic complexity of Block-Krylov PCA depends on poly( k √ ϵ ) . It is favorable for coarse accuracy ϵ (i.e. when ϵ = Θ(1) ) and small rank k . It is not suitable, however, for larger k . E.g., it can be the case in applications where we need to keep k = n/ 20 of the original dimensions. The same holds when higher accuracy is required, e.g. when ϵ = 1 / poly( n ) . Then the complexity of these methods is already higher than standard eigensolvers.

In finite precision, the landscape is even less clear. At the time of this submission, the only work related to low-rank approximation PCA that we are aware of with end-to-end bit complexity bounds is [107] for the approximation of matrix functions applied on vectors. They prove that the Lanczos method can be stably applied to compute a vector u such that the quantity ∥ Au ∥ approximates ∥ A ∥ . This is a backward-approximate solution for the top-1 singular vector. Concurrently with this work, and independently, [80] analyzed the bit complexity of Block-Krylov PCA and achieved similar bounds as ours, albeit with different techniques. However, [80] did not describe how to compute the condition number which is required in order to adjust the machine precision. Our analysis covers the computation of all the involved parameters.

For general invariant subspaces ( k > 1) , consider one of the simplest randomized low-rank approximation algorithms, often referred to as 'subspace iteration' [67] or 'simultaneous iteration' [106]. The algorithm first samples a matrix G ∈ R n × l with i.i.d standard normal elements, where l = Θ( k ) , and computes Y ← ( AA T ) q AG , where typically q ≈ log(min { m,n } ) . It then returns the Q factor from the economy QR factorization of Y . It can be shown (see e.g. Corollary 10.10 of [67]) that

The returned matrix has slightly more than k columns, often referred to as 'oversampling,' but, importantly, there is no explicit SVD involved in the computation. Even in this case, floating point arithmetic already spoils the approximation guarantees: we cannot compute Q without rounding errors. Stability analysis of QR factorization is typically carried out in the backward-error sense [70, 39], which makes the analysis of low-rank approximation PCA algorithms even more complicated.

G.3 Bit complexity analysis of Block-Krylov PCA

We now analyze the seminal Block-Krylov iteration algorithm of [106], which is listed in Algorithm 8 for convenience (in exact arithmetic). For the floating point analysis the following methodology is used. The main result is stated in Theorem 4.2. We shall denote by T MM ( X , q ) the cost of multiplying X with a dense matrix with q columns with a numerically stable multiplication algorithm, like the one of Theorem A.2. Formally, for the rest of this section, we assume the following subroutine.

Definition G.1 ( MMX ) . Let X ∈ R m × n be the input matrix of Algorithm 8. We assume a subroutine MMX ( B ) , which takes as input a matrix B with k columns. It returns a matrix C ∈ R m × r such that

in a total of T MMX ( k ) floating point operations, using at most O (log(1 / u )) bits of precision. We also assume MMX ⊤ ( B ) which approximates the product X ⊤ B with the same cost and approximation bounds.

The standard inner-product based algorithm as well as a block variant of the MM algorithm of Theorem A.2 satisfy this definition.

Step 1: Constructing the Block-Krylov matrix. We first observe that we can scale the matrix G in Algorithm 8 to have norm at most one, since this does not affect the Krylov basis. We assume that the (scaled) Gaussian matrix G in Algorithm 8 is given exactly. This is not realistic, but it greatly simplifies the analysis and it can be addressed by using the floating point Gaussian sampler of Definition D.1 to obtain a small error with high probability.

Given G and X , we can construct the Krylov matrix K using O ( q ) calls to MMX ( G ) . The first call X 1 ← MMX ( G ) returns a matrix X 1 = XG + E 1 where ∥ E 1 ∥ ≤ u · O (poly( mk )) ∥ X ∥∥ G ∥ ≤ u · O (poly( mk )) , since we assumed constant norms. If we keep performing multiplications recursively to build the Krylov matrix ∥ ˜ K ∥ , after O ( q ) multiplications it holds that ∥ ˜ K -K ∥ ≤ u · O (poly( mkq )) . The total cost is O ( qT MMX ( k )) floating point operations. If X is sparse and we use the standard inner product-based algorithm, T MMX ( k ) = O ( k nnz ( X )) . If X is dense and we use Theorem A.2 then O ( qmk ω -2 ) floating point operations are sufficient.

Step 2: Condition number of the Block-Krylov matrix. So far we have approximated K by ˜ K = K + E K .

In the following steps we will need an approximation for the condition number of K . We can get such an approximation using a variant of the COND algorithm (Corollary E.1), which internally uses

the SIGMAK algorithm of Proposition E.2 based on counting queries to approximate the condition number. In that case, we start with some arbitrary error ∥ E K ∥ , and we keep keep halving it until we have a sufficiently good approximation. At each iteration t , the algorithm executes

floating point operations to construct the Krylov matrix using O (polylog( m,q, 1 ϵ t ) bits. At most O (log( κ ( K ))) iterations are required. The most expensive iteration is the last one, when ϵ t ≈ 1 /κ ( K ) , which gives a total cost of at most

floating point operations using at most O (polylog( m,q,κ ( K ))) bits. It returns a value ˜ κ ∈ Θ( κ ( K )) , and it succeeds with high probability.

Step 3: Computing a basis. The computation of the basis in Step 4 of Algorithm 8 is arguably the hardest part in the analysis of the Block-Krylov iteration. With similar arguments as in the proof of Theorem A.6, if we write the economy-QR factorizations ˜ K = Q ˜ K R ˜ K and K = QR , there exists an orthogonal matrix Φ 1 such that Q -Q ˜ K Φ 1 = E 1 and ∥ E 1 ∥ ≤ ϵ ′ , for some ϵ ′ . This requires that ∥ ˜ K -K ∥ ≤ ϵ ′ 1 4 1 poly( mnqk ) κ ( K ) , which implies a requirement

for some constant c . Instead of κ ( K ) we can use ˜ κ from the previous Step 2. We can now use Theorem A.6 (which computes a basis via QR) on ˜ K to approximate Q ˜ K by the matrix

Let E Q = E 1 Φ 2 + E QR . We have that ∥ E Q ∥ = ∥ E 1 Φ 2 + E QR ∥ ≤ ϵ ′ + E QR . From Theorem A.6 we can ensure that ∥ E QR ∥ ≤ ϵ ′ if we use O (log( mqκ ( ˜ K ) ϵ ′ )) bits. Thus, ˜ Q is an approximate orthogonal basis for the range of the true Block-Krylov matrix K .

To summarize, as long as u ≤ ϵ ′ c poly( mq ) κ ( K ) , which can be achieved by replacing κ ( K ) with ˜ κ above, then we can compute ˜ Q = QΦ + E Q , where Q is the true orthonormal basis from the QR factorization of K , Φ is a qk × qk orthogonal matrix, and ∥ E Q ∥ ≤ ϵ ′ . The total cost is O ( m ( qk ) ω -1 ) floating point operations for Theorem A.6, using O ( log( mqκ ( K ) ϵ ′ ) ) bits.

Step 4: Computing the reduced matrix. In Step 5, Algorithm 8 forms the matrix M to compute its topk singular vectors. To analyze the computation of M , the first observation is that Q in lines 47 can be replaced by any basis for the column space of K . In particular, we replace it by ˜ Q . We then perform the multiplication in two steps: ˜ M ⊤ 1 = MMX ⊤ ( ˜ Q ) , which returns ˜ M 1 = ˜ Q ⊤ X + E MM 1 , where ∥ E MM 1 ∥ ≤ u · O (poly( qknm )) ∥ ˜ Q ∥∥ X ∥ ∈ u · O (poly( qm )) . Then we compute ˜ M ← MM ( ˜ M 1 , ˜ M ⊤ 1 ) = ˜ M 1 ˜ M ⊤ 1 + E MM 2 where ∥ E MM 2 ∥ ≤ u · O (poly( qkn )) ∥ M 1 ∥ 2 ∈ u · O (poly( qkn )) , where the last is implied if u is sufficiently smaller than 1 / poly( qm ) . Putting everything together we can write

where E M contains the errors in ˜ Q and also the ones from the multiplication in floating point, and ∥ E M ∥ ≤ O ( ∥ E Q ∥ ) ∈ O ( ϵ ′ ) . ϵ ′ is as in Step 3. So far we have assumed O (log( mqκ ( ˜ K ) ϵ ′ )) bits.

Step 5: Spectral gap and midpoint. Now we have written ˜ M = Φ ⊤ Q ⊤ XX ⊤ QΦ + E M , where Φ = Φ 2 Φ 1 is orthogonal and ∥ E M ∥ ≤ O ( ϵ ′ ) for some ϵ ′ ∈ (0 , 1) . We can use the counting queries in the spirit of Theorem 3.1 to compute the spectral gap and the midpoint of M .

In particular, we start with some desired bound ϵ ′ 0 ∈ (0 , 1) for ∥ E M ∥ by setting the number of bits to O (log( qmκ ( ˜ K ) ϵ ′ 0 )) . At each iteration we repeat Steps 1, 3, and 4 (we do not need to compute

again the condition number of K in Step 2). After at most m = O (log(1 / gap k ( M ))) iterations, the error ϵ ′ m satisfies ϵ ′ m = Θ(gap k ( M )) , and we obtain two quantities ˜ gap k ∈ (1 ± 1 8 ) gap k ( M ) and ˜ µ k ∈ µ k ( M ) + 1 8 gap k ( M ) with high probability. In each iteration we need to construct ˜ M using Steps 1, 3, and 4 with the specified number of bits.

Step 1 costs O ( qT MMX ( k )) operations. Step 3 requires O ( m ( qk ) ω -1 ) operations. Step 4 executes O ( T MMX ( qk ) + m ( qk ) ω -1 ) operations. The result is the qk × qk matrix ˜ M on which we call COUNT on each iteration. From Lemma E.1, assuming that we have regularized ˜ M appropriately using REGULARIZE , COUNT costs at most O (( qk ) ω polylog( qk/ϵ ′ m )) = O (( qk ) ω polylog( qk/ gap k ( M ))) . The maximum number of bits is in the last iteration, which is equal to O ( polylog ( mqκ ( K ) gap k ( M ) )) .

Step 6: Principal singular vectors. Given a suitable approximation for the gap and the midpoint, we next use Lemma B.1 to prove forward error bounds between the true k -spectral projector of M = Φ ⊤ Q ⊤ XX ⊤ QΦ and the one of ˜ M = M + E M . From Lemma B.1, if we set µ = ˜ µ k , we have that ∥ sgn( M -˜ µ k ) -sgn( ˜ M -˜ µ k ) ∥ ≤ ϵ SGN if ∥ E M ∥ ≤ ϵ SGN | λ min ( ˜ µ k -M ) | 2 π 128 = ϵ SGN Θ(gap k ( M ) 2 ) . The same holds for the spectral projectors, i.e. ∥ Π k ( M ) -Π k ( ˜ M ) ∥ ≤ ϵ SGN . The bound for ∥ E M ∥ is achieved by setting u ≤ ϵ SGN ˜ gap 2 k poly( mq ) ˜ κ , which translates to O ( log( mqκ ( K ) ϵ SGN ˜ gap k ) ) bits.

It remains to approximate Π k ( ˜ M ) , denoted as Π k for simplicity, and then use deflation, similar to the vanilla PCA Algorithm 7. We first set ϵ PURIFY = ( ϵ 2 PCA 20 3 ( qk ) 4 ) 2 , and then use Algorithm 3,

followed by ˜ U k ← DEFLATE ( ˜ Π k , k, ϵ PURIFY , ϵ PCA ) . From Proposition 2.1, PURIFY costs O (( qk ) ω log( qk/ gap k )) floating point operations using

bits. From Theorem A.5, DEFLATE costs O (( qk ) ω ) operations using O (polylog( qk/ϵ PCA )) bits, and succeeds with high probability. The returned matrix ˜ U k satisfies ∥ ˜ U k -U k ∥ ≤ ϵ PCA , where U k is a matrix whose columns form an orthonormal basis for the span of the topk singular vectors of ˜ M . This also implies that ∥ ˜ U k ˜ U ⊤ k -U k U ⊤ k ∥ ≤ 3 ϵ PCA . Then

We finally return ˜ Z k = MM ( ˜ Q , ˜ U k ) . It holds that where ∥ ˜ Z k -˜ Q ˜ U k ∥ ≤ u · O (poly( mq )) . We can then write

where QΦΠ k ( M ) Φ ⊤ Q ⊤ = Z k Z ⊤ k is equivalent to the true, exact arithmetic Z k of Algorithm 8. The error matrices satisfy ∥ E Z ∥ ≤ u · O (poly( mq )) , ∥ ˜ QE Π ˜ Q ⊤ ∥ ≤ O ( ϵ SGN + ϵ PCA ) , and ∥ E ′ ∥ ≤ O ( ∥ E Q ∥ ) ∈ O ( ∥ E M ∥ ) ∈ O ( ϵ SGN gap k ( M ) 2 ) .

Putting everything together. We can now state the main result by summarizing the Steps 1-6. Theorem G.2 (Restatement of Theorem 4.2) . Let X be a data matrix X ∈ R m × n , ∥ X ∥ ≤ 1 , k ∈ [ n ] a target rank, ϵ PCA ∈ (0 , 1) an accuracy parameter, and q = Θ ( log( n ) √ ϵ PCA ) . Let T MMX ( k ) denote the

complexity to stably multiply X or X ⊤ with a dense matrix with k columns from the right (see Def. G.1). Using the Steps 1-6 that are detailed in Appendix G.3 as a floating point implementation of Algorithm 8, we can compute a matrix ˜ Z k ∈ R m × k that satisfies

with high probability, where Z k is an approximate basis for the topk principal components of X , returned by Algorithm 8 in exact arithmetic. The total cost is at most

floating point operations, using O ( polylog( mqκ ( K ) ϵ PCA gap k ) ) bits of precision. K , M are the same as in Alg. 8. The only parameters that are initially required are k and ϵ PCA .

Proof. We first compute ˜ κ ∈ Θ( κ ( K )) in Steps 1 and 2. This costs

floating point operations using O (polylog( qmκ ( K ))) bits.

Then, in Step 5 we iteratively use Steps 1, 3, and 4 to compute the midpoint and the gap, in a total of O (log(1 / gap k )) iterations. The total cost is

floating point operations using at most

bits of precision. The only parameter that we require to know beforehand, except for the matrix sizes, is ˜ κ from the previous step.

In the last Step 6, assuming that ϵ SGN = ϵ PCA , we need to run Steps 1, 3, and 4, using O ( polylog( mqκ ( K ) ϵ PCA gap k ) ) bits. The number of arithmetic operations does not exceed the one from the previous step. Thereafter the costs of DEFLATE , PURIFY , and MM are negligible compared to the costs of Steps 1, 3, and 4. The final matrix ˜ Z k satisfies

where Z k is the true, exact arithmetic projector in Algorithm 8.

Summarizing everything, we can compute ˜ Z k as advertised above in a total of at most

floating point operations, using O ( polylog( mqκ ( K ) ϵ PCA gap k ) ) bits of precision.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: All claims accurately follow the paper contributions.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Limitations have been clearly discussed in the paper.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate 'Limitations' section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All theoretical results include an informal proof in the main paper and a reference to the full, rigorous proof in the appendix. All definitions and assumptions are clearly stated.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: There are no experiments, only theoretical results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: There are no experiments and no code associated with the paper.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: There are no experiments in the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer 'Yes' if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There is no apparant societal impact of the work performed as it only includes mathematical results.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: There is no data or models involved in the paper.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: The paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
The paper analyses complexity of generalized symmetric eigenvalue problem computation.

STRENGTHS:
- Solid analysis of the computations involved in generalized eigenvalue problems.
- Excellent survey of related works.
- Relevant applications in ML (PCA).

WEAKNESSES:
- The assumption of H being symmetric needs to be in the Abstract as well, as it is quite significant.
- Missing citation to kernel PCA [1]
- While the analysis is strong, the computational aspect is not as emphasized. For example, Algorithm 1 is not shown in main body and only in Appendix, and the paper has no experimental evaluation.

[1] Schölkopf, Bernhard, Alexander Smola, and Klaus-Robert Müller. "Nonlinear component analysis as a kernel eigenvalue problem." Neural computation 10.5 (1998): 1299-1319.

QUESTIONS:
My suggestion is to better emphasize the computational aspect and make the paper more suitable for a conference. For example, 
- Move the main Algorithms to the main body as it gives a more visual impact, which is of higher importance for a conference paper
- Use tables to compare accuracy and complexity of different algorithms, such as baselines like Lanczos, Randomized SVD, etc.
- It would be helpful if the authors could show numerical results w.r.t. accuracy, complexity, or relevant quantities to support their theoretical analysis.

LIMITATIONS:
Addressed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the following optimization problem: given Hermitian $H$ and Hermitian, positive definite $S$, find matrices $C$ and $\\Lambda$ such that $HC=SC\\Lambda$ where $C$ is the eigenvectors and $\\Lambda$ is the eigenvalues. The important application is when the eigenvectors of the interest form an invariant subspace, which covers applications such as DFT and PCA. So the goal is to compute a rank-$k$ projection onto the top or bottom-$k$ eigenvectors. The main contribution of this paper is a stable algorithm that computes an approximate projector $\\tilde \\Pi_k$ such that $\\|\\tilde \\Pi_k - \\Pi_k\\|\\leq \\epsilon$ where $\\Pi_k$ is the projector onto top/bottom-$k$ principal components. The algorithm runs in the current matrix multiplication time up to log factors in $n$, $\\kappa(S)$, $1/\\epsilon$  and the gap between $\\lambda_k, \\lambda_{k+1}$. It uses polylog bits of precision in the above parameters. It also provides error bounds for Cholesky beyond $O(n^3)$ classical algorithm. As corollaries, the algorithm leads to improvement for DFT, PCA and Block-PCA.\n\nSTRENGTHS:\nThis paper designs a novel algorithm for invariant subspace projection and PCA in the current matrix multiplication time. The algorithm is a good combination of existing techniques (such as approximating the sign function via inverse Newton) and new insights, such as reducing gap and midpoint of eigenvalues computation to eigenvalue threshold counting, and solve the counting via smoothed analysis. \n\nThe paper is well-written and results are presented clearly in the main body, some intuitions and proof sketches are provided for better understanding of the algorithms and analysis. Overall, I think this is a good paper with solid results.\n\nWEAKNESSES:\nThe only complaint I have with this paper (might be a bit unfair) is due to the sheer amount of contents and its very numerical linear algebra nature, this paper might be better suited for conferences such as ISSAC or journals such as SIAM journal on matrix analysis. Otherwise, I think this paper is definitely strong enough to be published in NeurIPS.\n\nQUESTIONS:\nFor your open question on rectangular FMM: is the current runtime bottleneck due to block square FMM? I didn't dig too deep into the algorithm details, but I would imagine the $n^\\omega$ bottleneck is due to certain linear algebraic operations on $n\\times n$ matrices, instead of a bunch of small rectangular FMMs?\n\nLIMITATIONS:\nAuthors have discussed limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

comments: SUMMARY:
The paper tackles the fundamental problem of GEP subspace approximation (with forward error approximation). The problem is very relevant to different areas of Machine learning. This paper improves the time complexity of this approximation from cubic in n to matrix multiplication time, which is a significant improvement. 

I have read the first nine pages and Appendix B, and the idea seems reasonable, and the paper is well explained. However, given the length of the paper (which exceeds 50 pages), I have not been able to verify the correctness of the solution. Therefore, I will not comment on the strengths and weaknesses of the paper and give a low confidence score. 

In my opinion, a more theoretical venue such as STOC/FOCS would be more appropriate for this paper (both because of the emphasis on theory and because the reviewers have significantly lower paper load). I wish the authors the best.

STRENGTHS:
-

WEAKNESSES:
-

QUESTIONS:
1. Could the authors please elaborate on the difference between their result and the forward error approximation in Theorem 5.2 of [1] (which is the [37] citation of the authors)? The authors of [1] write that their Theorem 5.2 is restrictive as it needs S to be invertible and H to have a simple eigenspectrum (no repeated eigenvalues). It seems that Theorem 1.1 by the authors also requires S to be invertible but can allow a non-simple spectrum. 

2. Is a general dependence on the term gap_k necessary for analysis like this?

LIMITATIONS:
The authors have adequately addressed the limitations of this work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel approach to approximating invariant subspaces of generalized eigenvalue problems (GEPs), which are fundamental in many applications such as Principal Component Analysis (PCA) and Density Functional Theory (DFT). The authors introduce an algorithm that computes a spectral projector $ \\Pi_k $ with a forward-error approximation guarantee in near matrix multiplication time $ O(n^{\\omega + \\eta}) $, where $ \\omega $ is the matrix multiplication exponent. The approach advances a new analysis for Cholesky factorization and a smoothed analysis for computing spectral gaps, which are key innovations applied to obtain the desired bound with high probability.\n\nThe paper's technical claims are well-supported by rigorous mathematical proofs and thorough analysis. The use of Cholesky factorization and smoothed analysis for spectral gaps is innovative and effectively addresses the computational challenges. The proposed algorithm’s performance is theoretically grounded. However, some sections could benefit from additional explanations, empirical computations and examples to enhance understanding, especially for readers less familiar with the theoretical work developed in the paper, and potentially reaching a larger audience.\n\nSTRENGTHS:\n- **Originality:** The paper introduces a novel approach to solving GEPs and PCA with nearly matrix multiplication time complexity, which is a significant advancement over classical methods. The novel analysis for Cholesky factorization and a smoothed analysis for computing spectral gaps could have a wider application too.\n- **Quality:** The mathematical rigor and comprehensive analysis in terms of bit complexity ensure that the proposed methods are both theoretically sound and practically applicable.\n- **Clarity:** The paper is generally well-written, with detailed explanations of the methods and thorough proofs.\n- **Significance:** The results have broad implications for many applications in machine learning and scientific computing, providing a more efficient computational framework.\n\nWEAKNESSES:\n- **Complexity of Implementation:** The proposed methods, while theoretically sound, may be complex to implement in practice. Detailed implementation guidelines for the algorithms 1, 2, 3, and 4. A standard implementation also would be of value, showing how some of the factors hidden in the $O(.)$ complexity could play a significant role in practice. For example, as it is, it remains open how, for example, the significance of the factor defining the bound on the approximation error $\\epsilon$ (which appears inside a $\\text{log} \\text{log}$ expression) when working with varied sizes of matrices $n$ and approximation error $\\epsilon$. Theorem 1 also should include some comments about this issue.\n- **Hyperparameter Sensitivity:** The algorithm's performance depends on several parameters (e.g., spectral gap, condition number). A more detailed discussion on selecting these parameters and their impact on performance would be helpful, along with empirical implementations and discussion.\n\nQUESTIONS:\n1) Parameter Selection: How should the hyperparameters (e.g., spectral gap, condition number) be chosen in practice? Are there heuristic methods or guidelines for selecting these parameters to ensure optimal performance?\n2) Extensions to Sparse Matrices: How does the proposed method handle sparse matrices common in large-scale applications? Are there any modifications or optimizations specific to sparse matrix structures?\n\nLIMITATIONS:\nThe presented paper discusses in great depth the theoretical underpinning of the proposed algorithm, and we understand this already has a significant impact. Nevertheless, the lack of practical implementation, empirical validation, and analysis of some aspects of the algorithms hinders the potentially higher impact of the proposed method. Even a limited empirical section, including investigations into the role of the hyperparameters and covering varied structures of matrices, would highly improve the paper's potential impact and readership.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the computational cost of invariant subspaces for generalized eigenvalue problems (GEPs) which is a fundamental computational problem with applications in machine learning and scientific computing. The authors propose a novel method that approximates the spectral projectors of GEPs, and give the first $\\tilde{O}(n^{\\omega})$ bit complexity result for forward-error approximation in the floating point model, improve upon the previous $\\tilde{O}(n^3)$ result. Based on this result, the authors also give new matrix multiplication-type upper bounds for PCA problems in terms of bit complexity.\n\nSTRENGTHS:\n1. This paper is well-written with very solid theoretical results and proofs. The problem, the model, the results are stated clearly. The existing works and previous results are discussed thoroughly.\n2. The idea of reducing the problem to approximating the average and the gap among two adjacent eigenvalues is interesting. The result of approximating these two quantities (Theorem 3.1) can have potential other applications.\n3. This paper gives a new stability analysis of the Cholesky factorization under floating point model and improves upon the previous $\\tilde{O}(n^3)$ result, which is of independent interest (especially in the TCS community).\n\nWEAKNESSES:\nThe main concern is that this paper may be too dense with technical details and rigorous proofs as a NeurIPS submission (instead of a TCS conference). It might be better if the authors consider presenting this paper in a way that is easier to follow (e.g., delay more proof details to appendix, while adding some examples of what the results look like for specific schemes of spectral decay, and more detailed discussion about the potential applications / examples of GEPs), so that this problem can be brought to more people in this community.\n\nQUESTIONS:\n1. In the statement of Theorem 1.1 and Proposition 2.1, is $||H|| \\leq 1$ a proper assumption?\n2. Also in Theorem 1.1, what if $gap_k$ is very small for the given $k$ (say, $gap_k = 2^{-n}$)?\n2. How does the result of Theorem 3.1 compare with previous works (if any)? How hard is it to approximate $\\mu_k$ and $gap_k$ for a given $k$, is there any previous work that also achieve the $\\tilde{O}(n^{\\omega})$ bit complexity result?\n3. Can the authors provide a more detailed discussion of how the proposed algorithm can (or cannot) be applied to real world problems such as solving large scale PCA tasks?\n\nLIMITATIONS:\nThe authors have adequately discussed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-19"}