Title: Oja's Algorithm for Streaming Sparse PCA
Venue: NeurIPS 2024
Accepted: yes
Keywords: Streaming PCA

Oja's Algorithm

Sparse PCA

Support Recovery

Entrywise Bounds
Abstract: Oja's algorithm for Streaming Principal Component Analysis (PCA) for $n$ data-points in a $d$ dimensional space achieves the same sin-squared error $O(r_{\mathsf{eff}}/n)$ as the offline algorithm in $O(d)$ space and $O(nd)$ time and a single pass through the datapoints. Here $r_{\mathsf{eff}}$ is the effective rank (ratio of the trace and the principal eigenvalue of the population covariance matrix $\Sigma$). Under this computational budget, we consider the problem of sparse PCA, where the principal eigenvector of $\Sigma$ is $s$-sparse, and $r_{\mathsf{eff}}$ can be large. In this setting, to our knowledge, *there are no known single-pass algorithms* that achieve the minimax error bound in $O(d)$ space and $O(nd)$ time without either requiring strong initialization conditions or assuming further structure (e.g., spiked) of the covariance matrix.
We show that a simple single-pass procedure that thresholds the output of Oja's algorithm (the Oja vector) can achieve the minimax error bound under some regularity conditions in $O(d)$ space and $O(nd)$ time.  
We present a nontrivial and novel analysis of the entries of the unnormalized Oja vector, which involves the projection of a product of independent random matrices on a random initial vector. This is completely different from previous analyses of Oja's algorithm and matrix products, which have been done when the $r_{\mathsf{eff}}$ is bounded.

Oja's Algorithm for Streaming Sparse PCA

Abstract

Oja's algorithm for Streaming Principal Component Analysis (PCA) for n datapoints in a d dimensional space achieves the same sin-squared error O ( r eff /n ) as the offline algorithm in O ( d ) space and O ( nd ) time and a single pass through the datapoints. Here r eff is the effective rank (ratio of the trace and the principal eigenvalue of the population covariance matrix Σ ). Under this computational budget, we consider the problem of sparse PCA, where the principal eigenvector of Σ is s -sparse, and r eff can be large. In this setting, to our knowledge, there are no known single-pass algorithms that achieve the minimax error bound in O ( d ) space and O ( nd ) time without either requiring strong initialization conditions or assuming further structure (e.g., spiked) of the covariance matrix. We show that a simple single-pass procedure that thresholds the output of Oja's algorithm (the Oja vector) can achieve the minimax error bound under some regularity conditions in O ( d ) space and O ( nd ) time. We present a nontrivial and novel analysis of the entries of the unnormalized Oja vector, which involves the projection of a product of independent random matrices on a random initial vector. This is completely different from previous analyses of Oja's algorithm and matrix products, which have been done when the r eff is bounded.

1 Introduction

Principal Component Analysis (PCA) [Pea01, Jol03] is a classical statistical method for data analysis and visualization. Given a dataset { X i } i =1 ,...,n where X i ∈ R d , sampled independently from a distribution with mean zero and covariance matrix Σ , the goal in PCA is to find the directions that explain most of the variance in the data. It is well known [Wed72, JJK + 16, Ver10] that the leading eigenvector, ˆ v , of the empirical covariance matrix, ˆ Σ , provides an optimal error rate under suitable tail conditions on the datapoints.

Computing ˆ v can be inefficient for large sample sizes, n , and dimensions d . Oja's algorithm [Oja82a] offers a comparable error rate in O ( nd ) time and O ( d ) space. Going back to the Canadian psychologist Donald Hebb's research [Heb49], it has attracted a lot of attention in theoretical Statistics and Computer Science communities [JJK + 16, AZL17, CYWZ18, YHW18, HW19a, HNW21, MP22, LSW21, Mon22, HNWW21]. In these works, the error metric is the sin 2 error between the estimated vector and the principal eigenvector of Σ (true population eigenvector v 1 ). Notably, [JJK + 16], [AZL17], and [HNW21] establish that Oja's algorithm achieves the same O ( r eff /n ) sin-squared error as the offline algorithm that estimates the top eigenvector of the empirical covariance matrix.

However, when the effective rank, r eff , of Σ (defined as Tr(Σ) / ‖ Σ ‖ ) is large, PCA has been shown to be inconsistent [Pau07, JM09, JL09]. This setting comes up in sparse PCA problems, when v 1 is s -sparse (i.e. has only s nonzero entries). Let ‖ . ‖ 0 denote the l 0 norm, i.e, the count of non-zero vector entries. Then, sparse PCA can be formally framed as the optimization problem:

Sin-squaredErrorwithTimesteps

(a)

(b)

Oja's Algorithm with Cardinality Truncation (Our Algorithm)

0.9

(  pue Bum) bus o   w s,eo

Stochastic Power Method with Truncation (Yang and Zu, 2015)

Diagonal Thresholding (Johnstone and Lu, 2009)

0.8

0.7

Sin-squared Error

0.6

0.5

0.4

0.3

0.2

0.1

0

0

200

400

600

800

1000

Timesteps

L

In general, without further assumptions, Problem (1) is non-convex and NP-hard [MWA06], as it reduces to subset selection in ordinary least squares regression.

[VL12, CMW13] showed a O ( σ 2 ∗ s log ( d ) /n ) minimax lower bound for the sin 2 error 1 -( v T 1 ̂ v ) 2 , where σ 2 ∗ := λ 1 λ 2 ( λ 1 -λ 2 ) 2 . Here λ 1 > λ 2 ≥ . . . λ d are the eigenvalues of Σ . Extensive research has been conducted on optimal offline algorithms for sparse PCA, some of which are convex relaxationbased [BR13, dBEG08, VCLR13, STL07, ZX18, DMMW17, AW08, Ma13, CMW13]. Others involve iterated thresholding [JNRS10, Ma13, YZ13], where a truncated power-method is analyzed along to achieve sparsity. For brevity, we only describe algorithms that fit within the computation budget in consideration, i.e., O ( nd ) time, O ( d ) space. For a detailed comparison, see Table 1 and Appendix Section A.1.

Support recovery algorithms in O ( nd ) time, O ( d ) space: Consider the spiked covariance model

where I d is the identity matrix, ν i > 0 , and v i are sparse. For the general case, we only assume v 1 is s -sparse. When r = 1 , Σ ii are the largest for i ∈ S . Diagonal thresholding essentially estimates Σ ii within our computational budget and uses thresholding to recover the support [JL09, AW08]. However, as we will show, without knowing the support sizes in each eigenvector and the number of spikes, this algorithm can fail, even in a spiked setting with r > 1 . Also, for r = 1 , [BPP18] show how to adapt a black-box algorithm for sparse linear regression for support recovery.

Sparse PCA algorithms in O ( nd ) time, O ( d ) space: The streaming sparse PCA algorithms proposed by [YX15] and [WL16] require an initialization u 0 with a sufficiently large | u T 0 v 1 | = Ω(1) (local convergence), which can be hard to find for large d and a general Σ . See Table 1 for details.

In light of this lack of O ( nd ) time, O ( d ) space globally convergent algorithms for sparse PCA, we ask the following question in this work:

Goal: Is there a single-pass algorithm that, under a general Σ with s -sparse v 1 , outputs ˆ v achieving the minimax sin 2 error ( 1 -〈 ̂ v, v 1 〉 2 ) with O ( d ) space, O ( nd ) time, without a strong initialization?

We provide a surprisingly simple answer to the above question:

Theorem 1.1 (Informal) . For a suitable range of the effective rank r eff and the ratio λ 1 /λ 2 , there exists a single pass algorithm A that recovers the support of v 1 using Oja's algorithm, operates under O ( d ) space, O ( nd ) time and returns ˆ v with the minimax optimal sin 2 error, O ( σ 2 ∗ s log ( d ) /n ) , for a general covariance matrix.

Our contributions:

Support recovery: We show, for a general Σ with the only constraint of a s -sparse v 0 that the top k entries of the Oja vector in magnitude include the true support with high probability. The Oja vector is initialized by a random unit vector.

Sparse PCA: We use the recovered support to achieve a minimax optimal sparse PCA algorithm.

Entrywise analysis: Our analysis is nontrivial and novel because it deviates from all existing analyses of matrix products and streaming PCA [HNWTW20, HW19b, LSW21, Lia23] which require ‖ X i ‖ 2 /λ 1 or r eff to be bounded to obtain the O (1 /n ) sin 2 error rate.

In Figure 1b, for a simple spiked model with r = 2 , we show the relative performances of all O ( nd ) time and O ( d ) algorithms in Table 1. Our thresholded and renormalized Oja algorithm outperforms all other algorithms operating under the same computational budget. The diagonal thresholding algorithm ( [JL09]), which is successful for the special case of r = 1 , has a large error in the general case.

We now present an outline of our paper. We start by describing the problem setup and assumptions in Section 2. Then we present our main results in Section 3, which includes our results for Support

Recovery (Section 3.1), Sparse PCA (Section 3.3) and Entrywise Deviation bounds (Section 3.5) for the Oja vector. Finally, we provide a sketch of the proof along with the techniques used in Section 4.

2 Problem setup and preliminaries

Notation . We use E [ . ] to denote expectation and [ n ] for { 1 , . . . , n } . The matrix multiplication constant is denoted as ω ≈ 2 . 372 . X ⊥ ⊥ Y represents statistical independence between random variables X and Y . The /lscript 2 norm for vectors and operator norm for matrices is ‖ . ‖ 2 , the count of nonzero vector elements ( /lscript 0 norm) is ‖ . ‖ 0 , and the Frobenius norm for matrices is ‖ . ‖ F . For v ∈ R d , R ⊆ [ d ] , /floorleft v /floorright R ∈ R d is the truncated vector with entries outside R set to 0. I d ∈ R d × d is the identity matrix, with i th column e i ∈ R d × 1 . For any set T ⊆ [ d ] , I T ∈ R d × d is defined as I T ( i, j ) = ✶ ( i, j ∈ T ) ✶ ( i = j ) , where ✶ ( . ) is the indicator random variable. 〈 A,B 〉 := Tr( A T B ) represents the matrix inner product. ˜ O and ˜ Ω represent order notations with logarithmic factors. We start by defining subgaussianity for multivariate distributions.

Definition 2.1. A random mean-zero vector X ∈ R d with covariance matrix Σ is a σ -subgaussian random vector ( σ > 0 ) if for all vectors v ∈ R d , we have E [ exp ( v T X )] ≤ exp ( σ 2 v T Σ v/ 2 ) . Equivalently, ∃ L > 0 , such that ∀ p ≥ 2 , ( E [ | v T X | p ]) 1 p ≤ Lσ √ p √ v T Σ v . 4

This definition of subgaussianity has been used in contemporary works on PCA and covariance estimation (See for example [MZ20, JLT20, DKPP23] and Theorem 4.7.1 in [Ver18]). We operate under the following two assumptions, unless otherwise specified,

Assumption 1 (Subgaussianity) . { X i } i ∈ [ n ] are of independent and identically distributed σ -subgaussian vectors in R d with covariance matrix Σ := E [ X i X T i ] .

We denote the eigenvectors of Σ as v 1 , v 2 , · · · v d and the corresponding eigenvalues as λ 1 > λ 2 ≥ · · · λ d . Define V ⊥ := [ v 2 , v 3 , · · · v d ] ∈ R d × ( d -1) and Λ 2 ∈ R ( d -1) × ( d -1) = diag ( λ 2 , λ 3 , · · · λ d ) .

/negationslash

Assumption 2 (Sparsity and Spectral gap) . We assume that max { 1 , λ 2 λ 1 -λ 2 } Tr(Λ 2 ) λ 1 -λ 2 ≤ cn log( n ) and λ 1 λ 1 -λ 2 ≤ c √ n log 2 ( n ) for an absolute constant c > 0 . The leading eigenvector, v 1 , satisfies ‖ v ‖ 0 ≤ s with support set S := { i : v 1 ( i ) = 0 } .

Remark 2.2. We note that Assumption 2 allows for r eff to be as large as d , given a sufficient eigengap. This can be observed by setting λ 1 = λ 2 (1 + g n ) for some g n > 0 . Note that Tr(Λ 2 ) λ 1 -λ 2 ≤ min ( 1+ g n g n r eff , 1 g n d ) . If g n ≤ 1 , then,

If g n /greatermuch 1 , then,

therefore, in both cases, as long as d ≤ ng n log n , r eff can be as large as d , while allowing for Assumption 2 to hold.

Oja's algorithm with constant learning rate. With a constant learning rate, η , and initial vector, u 0 , Oja's algorithm [Oja82b], denoted as Oja ( { X t } t ∈ [ n ] , η, u 0 ) , performs the updates, u t ← ( I + ηX t X T t ) u t -1 , u t ← u t ‖ u t ‖ 2 . For convenience of analysis, we also define ∀ t ∈ [ n ] ,

3 Main results

We present our main contributions in two stages. Firstly, in Section 3.1, we demonstrate that with an upper bound on the support size, the top elements of the Oja vector include the support with constant probability, which can be enhanced using a boosting procedure ( SuccessBoost ) described in Section 3.4. Secondly, in Section 3.3, we use the support to extract the eigenvector and provide a high-probability sin 2 error guarantee. Section 3.5 details our results on bounding the entrywise deviation of the Oja vector, which are crucial to our proofs and of independent interest. Detailed proofs are in the Appendix, Sections A.4 and A.5, with the learning rate, η , specified in Lemma A.2.4.

3.1 Support recovery

1: Input : Dataset { X i } i ∈ [ n ] , Cardinality parameter k ≥ s , learning rate η > 0

2: u 0 ∼ N (0 , I )

3: ˆ v ← Oja ( { X i } i ∈ [ n ] , η, y 0 )

4: ̂ S ← Indices of k largest values of | ˆ v |

5: return ̂ S

Algorithm 1 provides an estimate, ̂ S , of the true support set, S . It computes the Oja vector and returns the set of indices corresponding to its k largest entries in absolute value. Our key result in Lemma 3.1 discusses the recovery of the support set, S , for any k ≥ s , without requiring exact knowledge of the sparsity parameter s . Using Algorithm 1, it provides a set ̂ S ⊇ S with probability at least 0.9.

If k = s , i.e, the size of the support is exactly known, then we can improve the result of Lemma 3.1 to obtain an estimator, ̂ S , of the support set with high probability. Theorem 3.2 provides the corresponding guarantees. The SuccessBoost algorithm uses geometric aggregation on subsets returned from Algorithm 1 run on log(1 /δ )

disjoint subsets of the data and is described in Section 3.4. Theorem 3.2 (High probability support recovery) . Let Assumptions 1, 2 hold. For dataset D := { X i } i ∈ [ n ] , let A be the randomized algorithm which computes ̂ S ← OjaSupportRecovery ( { X i } i ∈ [ n ] , k, η ) , where η := 3 log( n ) n ( λ 1 -λ 2 ) and k = s . Then, for δ ∈ (0 , 1) , min i | v 1 ( i ) | = ˜ Ω ( λ 1 λ 1 -λ 2 ( d n 2 ) 1 4 ) , ˜ S ← SuccessBoost ( { X i } i ∈ [ n ] , A , δ ) satisfies,

For comparison, existing support recovery algorithms for general Σ are known for convex-relaxationbased algorithms like the SDP-based algorithm of [LV15]. These require a much larger computational budget than ours. In Section 3.3, we show how to use the s -agnostic support recovery in Lemma 3.1 to perform Sparse PCA and obtain a sin 2 error guarantee, where the final high probability error bound is obtained using a similar probability-boosting argument. For the learning rate, we follow the convention in related work ([BDF13, XHS + 18, JJK + 16, AZL17, HNWW21]) and choose the optimal value of the learning rate, which requires the knowledge of λ 1 -λ 2 . We believe an educated guess of η would lead to consistency at the cost of a suboptimal error bound.

3.2 Comparison with other support recovery algorithms

We note that (see Table 1), for the spiked model with r = 1 (Eq 2) [JL09] and [AW08] provide a diagonal thresholding algorithm for support recovery using O ( d ) space and O ( nd ) time. 5 To achieve a high-probability guarantee for the estimated support set, ̂ S , of the form P ( ̂ S = S ) ≥ 1 -δ , they

require (Proposition 1, [AW08]) n = Ω ( s 2 log ( d ) + log ( 1 δ )) . In comparison, Theorem 3.2 requires a larger sample size.

Remark 3.3. In practice, we will not know whether Σ is spiked or general. So, we can always augment our support recovery algorithm by taking a union of the support from the Oja vector and diagonal thresholding, still maintaining O ( nd ) time and O ( d ) space.

However, diagonal thresholding only works for the spiked model with a single spike, which our results do not require. It is easy to construct a Σ where the elements in the support of an eigenvector with a small eigenvalue have a larger magnitude than those of v 1 (Eq A.13). Here the diagonal thresholding method fails (see Figure 1a and Proposition 3.4). The explicit construction and the proof of Proposition 3.4 are available in the Appendix Section A.2. Figure 1 a) plots the sin 2 error due to different Sparse PCA algorithms operating in O ( d ) space and O ( nd ) time on such a covariance matrix, Σ , which is visualized in Figure 1 a).

Proposition 3.4 (Lower bound for diagonal thresholding) . Let Assumption 1 hold. For any diagonalthresholding algorithm, A , performing support recovery with sparsity parameter s such that n = Ω ( σ 4 s 2 log ( d ) ) , there exists a covariance matrix Σ with principal eigenvector, v 1 , ‖ v 1 ‖ 0 = s , such that, P (∣ ∣ S ⋂ S ∣ ∣ = 0 ) ≥ 1 -d -10 .

It may seem that if r in Eq 2 is small, and the sparsity parameters of each v i , i ≤ r are known, then diagonal thresholding would work. However, in general, r can be as large as d and the union of supports of v i can be [ d ] .

3.3 Sparse PCA

In this section, we describe our results for Sparse PCA, which use the support recovery guarantees developed in Section 3.1. For the results in this section, we split the dataset D := { X i } i ∈ [ n ] into two halves and estimate the support using the first half as ̂ S ← OjaSupportRecovery ( { X i } i ∈ [ n 2 ] , k, η := 3 log( n ) n ( λ 1 -λ 2 ) ) and input, k ≥ s . The second half of the samples are then used to compute the estimated sparse eigenvector. Algorithm 2 describes a general procedure for Sparse PCA given access to an estimated support set, ̂ S . We start with an intuitive procedure in Theorem 3.5, which runs Oja's algorithm on the data and then uses the support to truncate the estimated eigenvector.

Algorithm 2 TruncateOja ( { X i } i ∈ [ n ] , ̂ S, A , Θ )

1: Input : Dataset { X i } i ∈ [ n ] , , estimated support set ̂ S ⊆ [ d ] , Algorithm A , Parameters Θ

2:

3:

4:

ˆ

v

←A

v

ˆ

(

truncvec return

i

}

i

/floorleft

∈

v

[

n

/floorright

ˆ

∥

{

X

∥

/floorleft

ˆ

v

ˆ

v

/floorright truncvec

Theorem 3.5 (Vector Truncation) . Let Assumptions 1 and 2 hold and k ≥ s . For dataset D := { X i } i ∈ [ n ] and w 0 ∼ N (0 , I ) , let A be the randomized algorithm which computes ˆ v truncvec ← TruncateOja ( { X i } i ∈ ( n 2 ,n ] , ̂ S, Oja , { η, w 0 } ) , where η := 3 log( n ) n ( λ 1 -λ 2 ) . Then, for min i | v 1 ( i ) | = ˜ Ω ( ( d n 2 ) 1 8 ) , ˜ v ← SuccessBoost ( { X i } i ∈ [ n ] , A , d -10 ) satisfies,

with probability at least 1 -d -10 , where C ′′ ≥ 0 is an absolute constant.

Remark 3.6 (Limitation) . Existing inconsistency results on PCA [JL09] provide a threshold for signal strength ( λ 1 -λ 2 ) /λ 1 , below which, the principal eigenvector of ˆ Σ is asymptotically orthogonal to v 1 . We believe a similar result may hold for the Oja vector, which leads to the signal strength condition in Assumption 2.

]

ˆ

S

ˆ

∥

S

∥

,

Θ

2

)

Note that the rate obtained in Theorem 3.5 nearly matches the minimax lower bound proved in [VL12, CMW13], up to a factor of λ 2 λ 1 and log ( d ) and has optimal dependence on s , and n . A limitation of Algorithm 2 is that it uses the estimated support, ̂ S , at the very end after computing the estimated eigenvector to enhance the signal by truncation. Instead, one may run Oja's algorithm on datapoints restricted to the recovered support in the beginning.

To this end, we use the algorithm in [Lia23] (denoted by OptimalOja , see Proposition A.5.3) for subgaussian data, which uses an iteration-dependent sequence of step-sizes { η i } i ∈ [ n ] . We run Algorithm 2 with OptimalOja as the procedure to do sparse PCA. This leads to the minimax error rate, shown in Theorem 3.7. The high probability bounds in Theorem 3.5 and 3.7 both use the support recovery guarantees derived in Section 3.1 and the boosting procedure described in Section 3.4. Detailed proofs for both results can be found in Appendix Section A.5.

Theorem 3.7 (Data Truncation) . Let Assumptions 1 and 2 hold and k ≥ s . For dataset D := { X i } i ∈ [ n ] and w 0 ∼ N (0 , I ) , let A be the randomized algorithm which computes ˆ v truncvec ←

with probability at least 1 -d -10 , where C ′′ ≥ 0 is an absolute constant.

Remark 3.8. Algorithm 2, with both Oja and OptimalOja as input procedures require a simple initialization vector w 0 ∼ N (0 , I ) . In contrast, the block stochastic power method-based algorithm presented in [YX15] provides local convergence guarantees (see Theorem 1) requiring a block size of O ( s log ( d )) . They provide an initialization procedure, but the theoretical guarantees to achieve such an initialization require block size Ω( d ) . [YZ13] also require a close enough initialization. In the particular setting of a single-spiked covariance model, they require | w T 0 v 1 | = Ω(1) . In comparison for Algorithm 2, 3.7, it suffices to have | w T 0 v 1 |≥ δ √ e with probability at least 1 -δ (see Lemma A.2.1).

3.4 Probabilistic boosting

In this section, we describe a generic procedure for boosting the success probability of a given randomized algorithm, A (also see [KLL + 23]). If A satisfies Definition 3.9, then its probability can be boosted using this procedure. The formal guarantees of the boosting procedure are provided in Lemma 3.10 (proof in Appendix Section A.2). It divides the data evenly into log ( 1 δ ) buckets 6 , runs A on each bucket and aggregates the results via pairwise comparisons.

Definition 3.9. Let T be a set with metric ρ and A be a randomized algorithm which takes as input n i.i.d datapoints D := { X i } i ∈ [ d ] and possibly additional statistically independent parameters θ , and returns an estimate q ∈ T , which satisfies P ( ρ ( q, q ∗ ) ≥ /epsilon1 ) ≤ 1 3 for a fixed q ∗ ∈ T . Then, A is said to be a constant success oracle with parameters ( D,θ, T , ρ, q ∗ , /epsilon1 ) , denoted as A := ConstantSuccessOracle ( D,θ, T , ρ, q ∗ , /epsilon1 ) .

Algorithm 3 SuccessBoost ( { X i } i ∈ [ n ] , A , δ )

1: Input : Dataset D := { X i } i ∈ [ n ] , A := ConstantSuccessOracle ( D,θ, T , ρ, q ∗ , /epsilon1 ) , Required failure probability δ

2: Return : An estimate ˜ q ∈ T such that P ( ρ (˜ q, q ∗ ) ≤ 3 /epsilon1 ) ≥ 1 -δ

3: S ← 30 log ( 1 δ ) , B ← n/S

4: ∀ t ∈ [ S ] , q t ←A ( { X B ( t -1)+ i } i ∈ [ B ] , θ, T , ρ, q ∗ , /epsilon1 ) , C t ←{ t ′ ∈ [ S ] : ρ ( q t , q t ′ ) ≤ 2 /epsilon1 }

5: If ∃ q t such that |C t | /S ≥ 0 . 4 Return q t Else Return ⊥

Value

20

15

10

Q

5

-10

0

Entry in support (sample)

Entry outside support (sample)

Entry in support (population)

—Entry outside support (population)

100

200

300

400

500

Timesteps

(a)

(b)

50

BBI

45

VB.BV

40

(1 +nm))²2n

Jain et al. (2016)

35

30

Value

25

20

15

10

5

Q

0

50

100

150

200

250

300

350

400

450

500

Timesteps

Lemma 3.10 (Geometric Aggregation for Boosting) . Let A := ConstantSuccessOracle ( D,θ, T , ρ, q ∗ , /epsilon1 ) (Definition 3.9) for dataset D := { X i } i ∈ [ n ] . Then for δ ∈ (0 , 1) , ˜ q ← SuccessBoost ( { X i } i ∈ [ n ] , A , δ ) satisfies P ( ρ (˜ q, q ∗ ) ≤ 3 /epsilon1 ) ≥ 1 -δ .

3.5 Entrywise deviation of the Oja vector

To analyze the success probability of recovering the indices in S , we will define the following event, E := { S ⊆ ̂ S } . We now upper bound P ( E c ) . Define an element of the unnormalized Oja vector as r i := e T i B n u 0 , i ∈ [ d ] . Here u 0 ∼ N (0 , I ) is the initialization used in Algorithm 1. Observe that

or equivalently,

Therefore, for any fixed τ n > 0 , E c = ⇒ {∃ i ∈ S, | r i |≤ τ n } ⋃ {|{ i : i / ∈ S, | r i |≥ τ n }| > k -s } We will, therefore, be interested in the tail behavior of r i for i ∈ S and i /negationslash∈ S . Before presenting our theorems, we will use Figure 2 to emphasize the daunting nature of what we aim to prove. Consider the quantity E [ r i | u 0 ] = E [ e T i B n u 0 | u 0 ] . We use X = C ± ∆ to denote | X -C |≤ ∆ .

Thus, traditional wisdom would make us hope that the elements, r i , will concentrate around their respective expectations, whose absolute values are off by a ratio | v 1 ( i ) || u T 0 v 1 | exp( nη ( λ 1 -λ 2 )) .

However, Figure 2(a) shows that while the elements in the support seem close to their expectation, those not in support are, on average, much larger than their expectation. First, note that elementwise analysis of the Oja vector has not been done even in the low dimensional regime where r eff /n → 0 . In this regime, there is very recent related work for eigenvectors of the empirical covariance matrix ˆ Σ [AFW22] which are not applicable here. In the high-dimensional case, an analog can be drawn with elements ˆ Σ , which concentrate around their mean individually. Yet, ‖ ˆ Σ -Σ ‖ is not small. Thus, thresholding ˆ Σ obtains consistent estimates of Σ under sparsity assumptions [BL09, DM + 16, Nov23].

A similar principle is applied by [SSM11] where the eigenvector of ˆ Σ is truncated. They assume that n is fixed, and λ 1 /λ 2 = d α →∞ as d →∞ . In comparison, our analysis is about products of random matrices, not sums, and hence, completely different. We will show that ˆ v 1 ( i ) , even when i ∈ S , do not concentrate. But for a suitably chosen threshold, They are large with high probability,

600

700

800

900

1000

whereas those outside S are much lower with high probability. Proving this is also difficult because the analysis involves the concentration of the projection of a product of independent high dimensional matrices on some initial random vector. Lemma 3.11 establishes exactly that for elements in the support.

Lemma 3.11 (Tail bound in support) . Fix a δ ∈ (0 . 1 , 1) . Define the event G := { | v T 1 u 0 |≥ δ √ e } and threshold τ n := δ √ 2 e min i ∈ S | v 1 ( i ) | (1 + ηλ 1 ) n . Let the learning rate be set as in Lemma 3.1. Then, for an absolute constant C H > 0 ,

Our next result provides a bound for i / ∈ S .

Lemma 3.12 (Tail bound outside support) . Fix a δ ∈ (0 . 1 , 1) . Let the learning rate be set as in Lemma 3.1 and define the threshold τ n := δ √ 2 e min i ∈ S | v 1 ( i ) | (1 + ηλ 1 ) n . Then, for min i | v 1 ( i ) | = ˜ Ω ( λ 1 λ 1 -λ 2 ( d n 2 ) 1 4 ) and an absolute constant C T > 0 we have,

The proofs of Lemmas 3.11 and 3.12 are based on tail-bounds involving the second and fourth moments of r i := e T i B n u 0 . The details of obtaining the tail bounds are deferred to the Appendix Section A.3. The results developed in this section are used to analyze the support recovery and sin 2 error guarantees provided in Section 3.1 and 3.3. We provide a brief proof sketch in Section 4.

4 Proof technique

In this section, we outline the proof techniques for the entrywise deviation bounds in Lemmas 3.11 and 3.12. These bounds are crucial for analyzing both the support recovery results (Lemma 3.1 and Theorem 3.2) and the sparse PCA results (Theorems 3.5 and 3.7). The proof involves deriving bounds on the expectation and second moment of u T 0 B n UU T B n u 0 , where U ∈ R d × k is a fixed matrix and u 0 ∼ N (0 , I ) . It then applies Chebyshev's inequality to obtain the tail bound. For the proof sketch, we use U = e i , but we maintain general notation for broader applicability in Theorem 3.5. For our results, we also need to bound this quantity with U = I S (see Lemma A.5.1 for details). Our techniques to bound E [ u T 0 B n UU T B n u 0 ] are detailed in Section 4.1.

4.1 Solving a linear system of recursions

One can show that (see Lemma A.2.11 in Appendix),

We start by showing how to bound α n and β n . Before we dive into our techniques, we note that the analysis of Oja's algorithm [JJK + 16] in the non-sparse setting provides some tools that we could potentially use here. Using the recursion from Lemma 9 in [JJK + 16], we get

where V is a variance parameter defined as ∥ ∥ E [ ( A 1 -Σ)( A 1 -Σ) T ]∥ ∥ . Lemma A.2.3 shows that for σ -subgaussian X (definition 2.1),

This provides an upper bound on α n ≤ ∥ ∥ E [ B n UU T B T n ]∥ ∥ . While this bound is tight when r eff is bounded by a constant, in the high dimensional setting (Assumption 2) considered in this work, this bound is too loose. This is evident from Figure 2(B), which plots ∥ ∥ E [ B n UU T B T n ]∥ ∥ for U = I , along with the bound achieved using Eq 6, labeled as Jain et al. (2016) . Note that the plots are in

the log -scale so a difference in the slopes translates to a significant multiplicative difference. This warrants a more fine-grained analysis of α n .

Let us examine α n more closely to obtain a finer bound. Using the structure of the matrix product, B n , from Eq 3, we have:

Now, as a consequence of subgaussianity (see Lemma A.2.2), for K := (2 L 2 σ 2 ) 2 , with the CauchySchwartz inequality, the second term in the RHS can be bounded further using:

Therefore, using the above bound along with the eigen-decomposition Σ := λ 1 v 1 v T 1 + V ⊥ Λ 2 V T ⊥

,

Similarly, β n can also be upper bounded as follows:

Note that upper bounding and eliminating α n -1 or β n -1 from Eq 8, 7 respectively, would simplify the recursion but lead to a weaker bound as in Eq 6. Therefore, we solve Eq 7 and 8 as a system of linear recursions in α n and β n .

Estimating elements of P n , where P is the defined 2 × 2 matrix, is crucial. [Wil92] gives a compact expression for these elements using λ 1 ( P ) and λ 2 ( P ) . Under our assumptions, we have P 11 > P 22 . A naive upper bound on λ 1 ( P ) using Weyl's inequality [Die15] is 1 + 2 ηλ 1 + c 3 Tr(Σ) , similar to Eq 6. Since recursions like Eq 9 are common in our analysis, we provide a general solution in Lemma A.2.5 (detailed in the Appendix Section A.2).

An important consequence of this is that we now have the following bounds on α n for U = I :

which is much tighter than Eq 6 in our high-dimensional regime. Furthermore, observing Figure 2(b), we see that Eq 10 presents a much tighter upper bound, matching (1 + ηλ 1 ) 2 n up to constant factors. Recall that the bounds obtained in this section deal with α n , β n defined in Eq 5. A similar system of recursions can be obtained to get tight bounds on E [ ( v T 1 B T n UU T B n v 1 ) 2 ] and E [ Tr ( V T ⊥ B T n UU T B n V ⊥ ) 2 ] , details of which we defer to the Appendix in Lemmas A.2.9, A.2.10.

5 Conclusion

Oja's algorithm for streaming PCA has been extensively studied in the recent theoretical literature, typically assuming that ‖ X i ‖ 2 /λ 1 is bounded or a slowly growing covariance matrix effective rank r eff . This paper addresses the high-dimensional sparse PCA setting where the effective rank r eff can be as large as n/ log n while v 1 is s -sparse. In this context, while there has been a vast body of work that achieves minimax error bounds, we are unaware of any single-pass algorithm that works in O ( nd ) time, O ( d ) space, on a general Σ , without any strong initialization. Surprisingly, our thresholded estimator achieves the minimax error bound of O ( s log d/n ) , whereas the error rate of Oja's algorithm is O ( r eff /n ) . Empirically, the elements of the unnormalized Oja vector do not concentrate in this regime. Through an analysis that uncouples the projection of a product of independent random matrices on v 1 and its orthogonal subspace, we show that the entries of the Oja vector within the support of v 1 are large, while those outside are much smaller.

Acknowledgments and Disclosure of Funding

We gratefully acknowledge NSF grants 2217069, 2019844, and DMS 2109155. We thank Kevin Tian for his valuable insight on geometric aggregation and boosting and the anonymous reviewers for their valuable feedback.

References

[AFW22] Emmanuel Abbe, Jianqing Fan, and Kaizheng Wang. An lp theory of pca and spectral clustering. The Annals of Statistics , 50(4):2359-2385, 2022.

[AW08] Arash A Amini and Martin J Wainwright. High-dimensional analysis of semidefinite relaxations for sparse principal components. In 2008 IEEE international symposium on information theory , pages 2454-2458. IEEE, 2008.

[AZL17] Zeyuan Allen-Zhu and Yuanzhi Li. First efficient convergence for streaming k-pca: a global, gap-free, and near-optimal rate. In 2017 IEEE 58th Annual Symposium on Foundations of Computer Science (FOCS) , pages 487-492. IEEE, 2017.

[BB19] Matthew Brennan and Guy Bresler. Optimal average-case reductions to sparse pca: From weak assumptions to strong hardness. In Conference on Learning Theory , pages 469-470. PMLR, 2019.

[BDF13] Akshay Balsubramani, Sanjoy Dasgupta, and Yoav Freund. The fast convergence of incremental pca. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 26 , pages 3174-3182. Curran Associates, Inc., 2013.

[BKW20] Afonso S Bandeira, Dmitriy Kunisky, and Alexander S Wein. Computational hardness of certifying bounds on constrained pca problems. In 11th Innovations in Theoretical Computer Science Conference (ITCS 2020) , volume 151, page 78. Schloss DagstuhlLeibniz-Zentrum f¨ ur Informatik, 2020.

[BL09] Peter J. Bickel and Elizaveta Levina. Covariance regularization by thresholding. arXiv: Statistics Theory , 2009.

[BPP18] Guy Bresler, Sung Min Park, and Madalina Persu. Sparse pca from sparse linear regression. Advances in Neural Information Processing Systems , 31, 2018.

[BR13] Quentin Berthet and Philippe Rigollet. Optimal detection of sparse principal components in high dimension. 2013.

[CMW13] T Tony Cai, Zongming Ma, and Yihong Wu. Sparse pca: Optimal rates and adaptive estimation. 2013.

[CYWZ18] Minshuo Chen, Lin Yang, Mengdi Wang, and Tuo Zhao. Dimensionality reduction for stationary time series via stochastic nonconvex optimization. Advances in Neural Information Processing Systems , 31, 2018.

[dBEG08] Alexandre d'Aspremont, Francis Bach, and Laurent El Ghaoui. Optimal solutions for sparse principal component analysis. Journal of Machine Learning Research , 9(7), 2008.

[Die15] Rainer Dietmann. Weyl's inequality and systems of forms. Quarterly Journal of Mathematics , 66(1):97-110, 2015.

[DKPP23] Ilias Diakonikolas, Daniel Kane, Ankit Pensia, and Thanasis Pittas. Nearly-linear time and streaming algorithms for outlier-robust pca. In International Conference on Machine Learning , pages 7886-7921. PMLR, 2023.

[DKWB23] Yunzi Ding, Dmitriy Kunisky, Alexander S Wein, and Afonso S Bandeira. Subexponential-time algorithms for sparse pca. Foundations of Computational Mathematics , pages 1-50, 2023.

[DM + 16] Yash Deshp, Andrea Montanari, et al. Sparse pca via covariance thresholding. Journal of Machine Learning Research , 17(141):1-41, 2016.

[DMMW17] Santanu S. Dey, Rahul Mazumder, Marco Molinaro, and Guanyi Wang. Sparse principal component analysis and its l 1 -relaxation, 2017.

[GMZ17] Chao Gao, Zongming Ma, and Harrison H Zhou. Sparse cca: Adaptive estimation and computational barriers. 2017.

[GWS20] Milana Gataric, Tengyao Wang, and Richard J Samworth. Sparse principal component analysis via axis-aligned random projections. Journal of the Royal Statistical Society Series B: Statistical Methodology , 82(2):329-359, 2020.

[Heb49] Donald O. Hebb. The organization of behavior: A neuropsychological theory . Wiley, New York, June 1949.

[HJS + 22] Baihe Huang, Shunhua Jiang, Zhao Song, Runzhou Tao, and Ruizhe Zhang. Solving sdp faster: A robust ipm framework and efficient implementation. In 2022 IEEE 63rd Annual Symposium on Foundations of Computer Science (FOCS) , pages 233-244. IEEE, 2022.

[HNW21] De Huang, Jonathan Niles-Weed, and Rachel Ward. Streaming k-pca: Efficient guarantees for oja's algorithm, beyond rank-one updates. CoRR , abs/2102.03646, 2021.

[HNWTW20] De Huang, Jonathan Niles-Weed, Joel A. Tropp, and Rachel Ward. Matrix concentration for products, 2020.

[HNWW21] De Huang, Jonathan Niles-Weed, and Rachel Ward. Streaming k-pca: Efficient guarantees for oja's algorithm, beyond rank-one updates, 2021.

[HW19a] Amelia Henriksen and Rachel Ward. AdaOja: Adaptive Learning Rates for Streaming PCA. arXiv e-prints , page arXiv:1905.12115, May 2019.

[HW19b] Amelia Henriksen and Rachel Ward. Concentration inequalities for random matrix products. arXiv e-prints , page arXiv:1907.05833, July 2019.

[JJK + 16] Prateek Jain, Chi Jin, Sham Kakade, Praneeth Netrapalli, and Aaron Sidford. Streaming pca: Matching matrix bernstein and near-optimal finite sample guarantees for oja's algorithm. In Proceedings of The 29th Conference on Learning Theory (COLT) , June 2016.

[JKL + 20] Haotian Jiang, Tarun Kathuria, Yin Tat Lee, Swati Padmanabhan, and Zhao Song. A faster interior point method for semidefinite programming. In 2020 IEEE 61st annual symposium on foundations of computer science (FOCS) , pages 910-918. IEEE, 2020.

[JKL + 24] Arun Jambulapati, Syamantak Kumar, Jerry Li, Shourya Pandey, Ankit Pensia, and Kevin Tian. Black-box k-to-1-pca reductions: Theory and applications. In Shipra Agrawal and Aaron Roth, editors, Proceedings of Thirty Seventh Conference on Learning Theory , volume 247 of Proceedings of Machine Learning Research , pages 2564-2607. PMLR, 30 Jun-03 Jul 2024.

[JL09] Iain M. Johnstone and Arthur Yu Lu. On consistency and sparsity for principal components analysis in high dimensions. Journal of the American Statistical Association , 104(486):682-693, 2009. PMID: 20617121.

[JLT20] Arun Jambulapati, Jerry Li, and Kevin Tian. Robust sub-gaussian principal component analysis and width-independent schatten packing. Advances in Neural Information Processing Systems , 33:15689-15701, 2020.

[JM09] Sungkyu Jung and J. S. Marron. PCA consistency in high dimension, low sample size context. The Annals of Statistics , 37(6B):4104 - 4130, 2009.

[JNRS10] Michel Journ´ ee, Yurii Nesterov, Peter Richt´ arik, and Rodolphe Sepulchre. Generalized power method for sparse principal component analysis. J. Mach. Learn. Res. , 11:517-553, mar 2010.

[Jol03] Ian T Jolliffe. Principal component analysis. Technometrics , 45(3):276, 2003.

[KLL + 23] Jonathan Kelner, Jerry Li, Allen X Liu, Aaron Sidford, and Kevin Tian. Semi-random sparse recovery in nearly-linear time. In The Thirty Sixth Annual Conference on Learning Theory , pages 2352-2398. PMLR, 2023.

[Lia23] Xin Liang. On the optimality of the oja's algorithm for online pca. Statistics and Computing , 33(3):62, 2023.

[LSH22] Hanbyul Lee, Qifan Song, and Jean Honorio. Support recovery in sparse pca with incomplete data. Advances in Neural Information Processing Systems , 35:2732127332, 2022.

[LSW21] Robert Lunde, Purnamrita Sarkar, and Rachel Ward. Bootstrapping the error of oja's algorithm. Advances in Neural Information Processing Systems , 34:6240-6252, 2021.

[LV15] Jing Lei and Vincent Q Vu. Sparsistency and agnostic inference in sparse pca. 2015.

[Ma13] Zongming Ma. Sparse principal component analysis and iterative thresholding. The Annals of Statistics , 41(2):772 - 801, 2013.

[Mac08] Lester Mackey. Deflation methods for sparse pca. Advances in neural information processing systems , 21, 2008.

[Mon22] Jean-Marie Monnez. Stochastic approximation of eigenvectors and eigenvalues of the q-symmetric expectation of a random matrix. Communications in Statistics-Theory and Methods , pages 1-15, 2022.

[MP22] Nikos Mouzakis and Eric Price. Spectral guarantees for adversarial streaming pca, 2022.

[MWA06] Baback Moghaddam, Yair Weiss, and Shai Avidan. Generalized spectral bounds for sparse lda. In Proceedings of the 23rd international conference on Machine learning , pages 641-648, 2006.

[MZ20] Shahar Mendelson and Nikita Zhivotovskiy. Robust covariance estimation under l 4-l 2 norm equivalence. 2020.

[Nov23] Gleb Novikov. Sparse pca beyond covariance thresholding. arXiv preprint arXiv:2302.10158 , 2023.

[Oja82a] Erkki Oja. Simplified neuron model as a principal component analyzer. Journal of Mathematical Biology , 15(3):267-273, November 1982.

[Oja82b] Erkki Oja. Simplified neuron model as a principal component analyzer. Journal of mathematical biology , 15:267-273, 1982.

[Pau07] Debashis Paul. Asymptotics of sample eigenstructure for a large dimensional spiked covariance model. Statistica Sinica , pages 1617-1642, 2007.

[Pea01] Karl Pearson. Liii. on lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science , 2(11):559-572, 1901.

[QLR19] Yixuan Qiu, Jing Lei, and Kathryn Roeder. Gradient-based sparse principal component analysis with extensions to online learning, 2019.

[SSM11] Dan Shen, Haipeng Shen, and J. S. Marron. Consistency of sparse pca in high dimension, low sample size contexts. J. Multivar. Anal. , 115:317-333, 2011.

[SSM13] Dan Shen, Haipeng Shen, and James Stephen Marron. Consistency of sparse pca in high dimension, low sample size contexts. Journal of Multivariate Analysis , 115:317-333, 2013.

[STL07] Bharath K. Sriperumbudur, David A. Torres, and Gert R. G. Lanckriet. Sparse eigen methods by d.c. programming. In Proceedings of the 24th International Conference on Machine Learning , ICML '07, page 831-838, New York, NY, USA, 2007. Association for Computing Machinery.

[VCLR13] Vincent Q Vu, Juhee Cho, Jing Lei, and Karl Rohe. Fantope projection and selection: Anear-optimal convex relaxation of sparse pca. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems , volume 26. Curran Associates, Inc., 2013.

[Ver10] Roman Vershynin. Introduction to the non-asymptotic analysis of random matrices. arXiv preprint arXiv:1011.3027 , 2010.

[Ver18] Roman Vershynin. High-Dimensional Probability . Cambridge University Press, Cambridge, UK, 2018.

[VL12] Vincent Vu and Jing Lei. Minimax rates of estimation for sparse pca in high dimensions. In Neil D. Lawrence and Mark Girolami, editors, Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics , volume 22 of Proceedings of Machine Learning Research , pages 1278-1286, La Palma, Canary Islands, 21-23 Apr 2012. PMLR.

[Wai19] Martin J Wainwright. High-dimensional statistics: A non-asymptotic viewpoint , volume 48. Cambridge university press, 2019.

[Wed72] Per- ˚ Ake Wedin. Perturbation bounds in connection with singular value decomposition. BIT Numerical Mathematics , 12:99-111, 1972.

[Wil92] Kenneth S. Williams. The nth power of a 2 × 2 matrix. Mathematics Magazine , 65(5):336-336, 1992.

[WL16] Chuang Wang and Yue M. Lu. Online learning for sparse pca in high dimensions: Exact dynamics and phase transitions, 2016.

[XHS + 18] Peng Xu, Bryan He, Christopher De Sa, Ioannis Mitliagkas, and Chris Re. Accelerated stochastic iteration. volume 84 of Proceedings of Machine Learning Research , pages 58-67, Playa Blanca, Lanzarote, Canary Islands, 09-11 Apr 2018. PMLR.

[YHW18] Puyudi Yang, Cho-Jui Hsieh, and Jane-Ling Wang. History pca: A new algorithm for streaming pca. arXiv preprint arXiv:1802.05447 , 2018.

[YX15] Wenzhuo Yang and Huan Xu. Streaming sparse principal component analysis. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning , volume 37 of Proceedings of Machine Learning Research , pages 494-503, Lille, France, 07-09 Jul 2015. PMLR.

[YZ13] Xiao-Tong Yuan and Tong Zhang. Truncated power method for sparse eigenvalue problems. Journal of Machine Learning Research , 14(4), 2013.

[ZX18] Hui Zou and Lingzhou Xue. A selective overview of sparse principal component analysis. Proceedings of the IEEE , 106(8):1311-1320, 2018.

A Appendix

The Appendix is organized as follows:

Section A.1 provides further details about related work

Section A.2 provides some useful results used in subsequent analyses

Section A.3 provides Entrywise deviation bounds for the Oja vector (Lemmas 3.11, 3.12)

Section A.4 proves convergence of Support Recovery results (Lemma 3.1, Theorem 3.2)

Section A.5 proves convergence of Sparse PCA results (Theorems 3.5,3.7)

Section A.6 provides another alternative way of truncation using a value-based thresholding (Theorem A.6.1)

A.1 Further details on related work

There has been a lot of work on computational computational hardness of sparse PCA [GMZ17, BB19, DKWB23, BKW20].

Minimax optimal Sparse PCA algorithms with global convergence: These consist of SDP-based algorithms such as [AW08, VCLR13, dBEG08], which do not scale well in high-dimensions (see [BR13, Wai19]). The state-of-the-art SDP solvers [JKL + 20, HJS + 22] currently have a runtime Ω( n ω + d ω ) , where ω ≈ 2 . 732 is the matrix multiplication exponent. Algorithms proposed in [Ma13, CMW13, JNRS10, DM + 16] involve forming the entire ( d × d ) sample covariance matrix, which can itself be challenging from the perspective of space and time complexity. Furthermore, [Ma13, CMW13, DM + 16] have been analyzed under the spiked covariance model in Eq 2. [QLR19] propose a computationally efficient modification of the Fantope projection-based algorithm of [VCLR13], which requires O ( d 2 ) space, and Ω( nd 2 ) time.

Single-pass online sparse PCA algorithms with O ( d 2 ) storage and O ( nd 2 ) time [QLR19] also provide a single-pass online algorithm and state that this algorithm (Theorem 4) is the first to provably obtain the global optima in a streaming setting without any initialization, under a general Σ . However, this method requires O ( d 2 ) storage, O ( nd 2 ) time, and the estimation error is O ( d 2 √ n ) (Theorem 4, [QLR19]). The algorithm does d sparse linear regression problems to achieve this.

Support recovery algorithms with O ( d 2 ) storage and O ( nd 2 ) time : [LV15, LSH22] use an SDP-based approach and [BPP18] use sparse linear regression for support recovery.

More details on streaming PCA algorithms [YX15] provides an online block version of the truncated power method in [YZ13] under the spiked model (Eq 2). They require an initialization u 0 with a sufficiently large | u T 0 v 1 | = Ω(1) (local convergence). Their proposed initialization with streaming PCA algorithm until reaching a specific accuracy threshold, for which there is no known theoretical guarantee under the spiked high-dimensional setting. [WL16] provides an analysis of streaming sparse PCA under Eq 2 via partial differential equations (PDE), but they only prove asymptotic convergence. Similar to [YZ13], they also require | u T 0 v 1 | = Ω(1) which can be hard to find in high dimensions for a general Σ . Recent results provide a black-box way to obtain the topk principal components ( k -PCA) given an algorithm to extract the top eigenvector (see [a]) which could be employed treating our algorithm as a 1-PCA oracle (see [JKL + 24, Mac08]). We believe that our analysis can be extended to obtain top-k principal components simultaneously via QR decomposition and thresholding.

A.2 Useful results

Lemma A.2.1. (Fact 2.9 [DKPP23]) For any symmetric d × d matrix A, we have Var z ∼N (0 ,I ) [ z T Az ] = 2 ‖ A ‖ 2 F . If A is a PSD matrix, then for any β > 0 , it holds that

Proof. We give a short proof here. Since A is a symmetric matrix, let A = P Λ P T where P is an orthonormal matrix and Λ is a diagonal matrix. Then, denoting y := P T z we note that y ∼ N (0 , I ) .

Therefore,

Therefore, and

/negationslash

To get the tail lower bound, note that it trivially follows if β > 1 . Therefore we proceed with β ∈ (0 , 1) . We have

Hence proved.

Lemma A.2.2. Let X ∈ R d be a σ -subgaussian random vector with covariance matrix Σ . Then, for any matrix M ∈ R d × m and any positive integer p ≥ 2 ,

Proof. Let the eigendecomposition of MM T be P Λ P T . Define Y := P T X . Then,

/negationslash

Therefore,

Using the definition of sub-gaussianity (Definition 2.1) we have,

Susbstituting in Eq A.12 we have,

Substituting in Eq A.11 we have,

Lemma A.2.3. Let X ∈ R d be a σ -subgaussian random vector with covariance matrix Σ . Then,

Proof. For any fixed unit vector u ∈ R d , we have

where we used Lemma A.2.2 with p = 2 and M = I .

Proposition 3.4 (Lower bound for diagonal thresholding) . Let Assumption 1 hold. For any diagonalthresholding algorithm, A , performing support recovery with sparsity parameter s such that n = Ω ( σ 4 s 2 log ( d ) ) , there exists a covariance matrix Σ with principal eigenvector, v 1 , ‖ v 1 ‖ 0 = s , such that, P (∣ ∣ S ⋂ S ∣ ∣ = 0 ) ≥ 1 -d -10 .

Proof. Let s be a multiple of 3 for ease of analysis. Consider a dataset with a covariance matrix,

where β 1 = 1 2 . Based on Eq A.13, we have for,

Note that the largest eigenvalue of Σ , λ 1 = β 1 + 1 2 . Let t n := 10 σ 2 λ 1 √ log( d ) n . Using Lemma 6.26 from [Wai19], we have, for the empirical covariance matrix, ˆ Σ ,

Define the event, E := max i,j ∈ [ d ] ∣ ∣ ∣ ˆ Σ( i, j ) -Σ( i, j ) ∣ ∣ ∣ ≤ t n and note that due to the sample complexity bound on n , under event E ,

Therefore, under event E , the s largest diagonal entries of ˆ Σ are i ∈ ( s, 2 s ] , and therefore, | ˆ S ⋂ S | = 0 , which completes our proof.

Lemma 3.10 (Geometric Aggregation for Boosting) . Let A := ConstantSuccessOracle ( D,θ, T , ρ, q ∗ , /epsilon1 ) (Definition 3.9) for dataset D := { X i } i ∈ [ n ] . Then for δ ∈ (0 , 1) , ˜ q ← SuccessBoost ( { X i } i ∈ [ n ] , A , δ ) satisfies P ( ρ (˜ q, q ∗ ) ≤ 3 /epsilon1 ) ≥ 1 -δ .

Proof. Consider the indicator random variables χ i := ✶ ( ρ ( q i , q ∗ ) ≤ /epsilon1 ) . Let p := 1 3 and r = 300 log ( 1 δ ) for convenience of notation. Then, ∀ i ∈ [ r ] , P ( χ i = 1) ≥ 1 -p . Define the set S := { i : i ∈ [ r ] , χ i = 1 } . We note that using standard Chernoff bounds for sums of independent Bernoulli random variables, for θ ∈ (0 , 1) ,

We have, E [ |S| ] ≥ r (1 -p ) using linearity of expectation. Therefore,

Recall that Algorithm 3 defines ˜ q as:

/negationslash

Note that the definition of ˜ q does not require knowledge of q ∗ and it can be computed by calculating ρ ( . ) error between all distinct ( r 2 ) pairs ( q i , q j ) i,j ∈ [ r ] ,i = j .

Let E be the event {|S| > 0 . 9 (1 -p ) r } and denote f := 0 . 9 (1 -p ) for convenience of notation. Let us now operate conditioned on E . Note that conditioned on E , such a ˜ q always exists since any point in S is a valid selection of ˜ q . This is true since

Here we used the property of the event E and the triangle inequality for ρ . Wefurther have, conditioned on E using triangle inequality for some i ∈ S ,

Therefore, we have

which completes our proof.

Lemma A.2.4 (Learning rate schedule) . Let the learning rate be set as η := κ log ( n ) n ( λ 1 -λ 2 ) for a positive constant κ > 0 . For constant c ≤ 1 8 κ min { 1 √ C 1 C } , let

If κ ≥ 2 + o (1) , n = Ω ( s 2 log ( d ) ) , the following hold:

where C := 100 ( L 4 σ 4 + L 2 σ 2 ) +16 . We state another useful restatement of Claim (1) used in subsequent analysis,

Proof.

Therefore, the first claim follows for c ≤ 1 κC . For the second claim,

where the last inequality holds for c ≤ 1 4 κC and n ≥ 1 . Furthermore we have

where the last inequality holds for κCc ≤ 1 4 . Note that ηC ≤ min { 1 4 λ 1 , 1 4 1 Tr(Λ 2 ) } imply 4 ηC ≤ 1 √ λ 1 Tr(Λ 2 ) .

For the third claim, we have

where the last inequality holds when c ≤ 1 √ 4 κ 2 C .

Next, we have the last claim,

Therefore, it suffices to ensure

where ( iv ) follows since λ 1 ( λ 1 -λ 2 ) ≥ 1 as λ 1 > λ 2 . Therefore, we require

which holds for κ = 2 + o (1) and sufficiently large n .

Lemma A.2.5. For constants c 1 , c 2 , c 3 , c 4 , c 5 > 0 , consider the following system of recursions -

Let ∃ θ ∈ (0 . 5 , 1) , which satisfies c 1 (1 -θ ) ( λ 1 -λ 2 ) + c 2 ηλ 2 1 = c 4 ηλ 2 Tr(Σ) and

Then we have,

where

Proof. Writing the recursions in a matrix form, we have

Define

Then P := I + c 1 ηM , where

and u := c 2 c 1 , v = c 3 c 1 , x = c 4 c 1 , w = c 5 c 1 . We now compute eigenvalues of M . The trace and determinants are given as -

Next we compute T 2 4 -D ,

Let ( λ 1 -λ 2 ) -xηλ 2 Tr(Σ) + ηuλ 2 1 = θ ( λ 1 -λ 2 ) for θ ∈ (0 , 1) ,

Let η 2 vwλ 2 1 λ 2 Tr(Σ) θ 2 ( λ 1 -λ 2 ) 2 ≤ 1 4 . Then, using the identity 1 -x 2 ≤ √ 1 + x ≤ 1 + x 2 for x ∈ (0 , 1) we have,

Let us simplify η 2 vwλ 2 1 λ 2 Tr(Σ) θ 2 ( λ 1 -λ 2 ) 2 using the definition of θ . We have

Let (1 -θ ) ≥ 4 ηuλ 2 1 ( λ 1 -λ 2 ) . Then,

Then, using Eq A.18 and A.19, the eigenvalues of M are given as λ 1 ( M ) := T 2 + √ T 2 4 -D and λ 2 ( M ) := T 2 -√ T 2 4 -D such that

The eigenvalues of P are given as λ 1 ( P ) := 1 + c 1 ηλ 1 ( M ) and λ 2 ( P ) := 1 + c 1 ηλ 2 ( M ) . Then we have,

We then use the result from [Wil92] to compute P n and α n , β n . To compute P n , we first compute the matrices X and Y -

Then, P n = λ 1 ( P ) n X + λ 2 ( P ) n Y , which gives

where

Therefore, for y 0 = [ α 0 β 0 ] , we have

Therefore, using Eq A.20 and Eq A.21,

Recall that using Eq A.20 and Eq A.21

Substituting in Eq A.24 and Eq A.25 we have,

Hence proved.

Lemma A.2.6. Let U ∈ R d × m then, for all t > 0 , under subgaussianity (Definition 2.1) and the step-size η satisfying (1 -θ ) ( λ 1 -λ 2 ) + 2 L 4 σ 4 ηλ 2 1 = 2 L 4 σ 4 ηλ 2 Tr(Σ) for θ ∈ ( 1 2 , 1 ) then we have

where B n is defined in Eq 3, α 0 = λ 1 v T 1 UU T v 1 , β 0 = Tr ( U T V ⊥ Λ 2 V T ⊥ U ) and

Proof. Let α n := λ 1 E [ Tr ( v T 1 B n UU T B T n v 1 )] , β n := E [ Tr ( U T B T n V ⊥ Λ 2 V T ⊥ B n U )] such that

Define A n := X n X T n and let F n denote the filtration for observations i ∈ [ n ] . Then,

α

n

=

=

=

=

=

=

=

=

1

λ

λ

1

α

E

E

n

α

α

α

α

α

α

α

[

[

-

n

n

n

n

n

n

1

-

1

-

1

-

1

-

1

-

1

-

1

T

1

v

B

v

T

1

n

(

UU

I

+2

+

T

B

ηA

ηλ

1

E

+2

T

n

n

)

[

v

1

ηλ

E

(1 + 2

(1 + 2

(1 + 2

(1 + 2

(1 + 2

n

1

-

(1 + 2

=

[

1

ηλ

ηλ

ηλ

1

1

ηλ

1

ηλ

1

ηλ

1 + 2

1

+4

ηλ

(

and similarly,

Writing the recursions in a matrix form, we have

The result then follows by using Lemma A.2.5.

Lemma A.2.7. Let U ∈ R d × m then, for all t > 0 , under subgaussianity (Definition 2.1) and the step-size η satisfying (1 -θ ) ( λ 1 -λ 2 ) + 2 L 4 σ 4 ηλ 2 1 = 2 L 4 σ 4 ηλ 2 Tr(Σ) for θ ∈ ( 1 2 , 1 ) then we have

1

η

v

B

T

1

1

]

n

A

v

T

1

-

n

1

UU

B

Σ

) +

n

T

B

-

B

η

) +

) +

n

1

UU

-

2

η

η

) +

η

) + 4

) + 4

2

4

L

σ

λ

2

2

1

UU

1

E

[

E

λ

1

2

λ

η

T

-

n

T

T

[(

v

[

E

T

1

(

B

B

T

n

X

-

n

T

1

v

1

1

T

n

-

(

I

1

v

X

X

[√

E

1

E

2

L

η

4

2

σ

4

L

4

λ

2

1

)

E

[√

E

4

λ

σ

4

α

λ

n

(

1

Tr

2

1

-

1

+

ηA

v

1

]

1

T

n

]

v

X

n

[

(

[

1

T

n

v

T

v

1

X

X

n

T

(

Σ

(

E

[

n

+

+

)

η

η

v

1

2

λ

2

) (

λ

X

1

T

n

) (

n

v

X

1

, v

Tr

+4

η

1

(

2

]

1

E

1

E

B

X

T

n

v

1

T

v

[

[(

v

T

1

A

v

-

n

T

n

v

1

X

T

1

λ

4

L

1

v

σ

n

)

E

1

v

4

λ

T

1

2

1

F

n

F

B

)

)

2

1

n

2

T

1

n

X

n

UU

-

∣

∣

∣

∣

∣

∣

∣

∣

[

Tr

+

β

(

Σ

V

n

-

⊥

1

T

1

UU

B

n

X

B

T

]

-

1

n

-

]

1

E

, B

Λ

n

2

V

-

T

⊥

-

T

n

T

n

-

B

T

n

[

X

-

(

[

(

n

X

1

UU

, B

1

1

v

UU

1

1

T

) (

X

X

n

)]

X

T

n

n

B

T

T

-

n

n

B

n

B

T

n

-

T

n

B

∣

∣

)

∣

∣

-

1

-

B

1

T

-

n

UU

F

1

A

n

-

n

-

n

v

1

1

UU

]]

1

UU

1

UU

1

)]

,

T

B

T

n

-

T

B

n

T

T

B

n

]

T

B

-

1

-

1

T

n

-

X

n

X

n

1

)

)

X

n

∣

∣

∣

∣

∣

2

2

] ]

1

] ]

∣

∣

F

F

n

)]

-

n

-

1

∣

using Lemma A.2.2 with

1

)])

(A.26)

E

p

= 2

where B n is defined in Eq 3, α 0 = v T 1 UU T v 1 , β 0 = Tr ( V T ⊥ UU T V ⊥ ) and

Proof. Let α n := E [ Tr ( v T 1 B n UU T B T n v 1 )] , β n := E [ Tr ( V T ⊥ B n UU T B T n V ⊥ )] . Define A n := X n X T n and let F n denote the filtration for observations i ∈ [ n ] . Then,

and similarly,

The result then follows by using Lemma A.2.5.

Lemma A.2.8. For all t > 0 , under subgaussianity (Definition 2.1), let U ∈ R d × m . Let the step-size η be set according to Lemma A.2.4 then we have,

where B n is defined in Eq 3, α 0 = v T 1 UU T v 1 , β 0 = Tr ( V T ⊥ UU T V ⊥ ) and

Proof. Let α n := E [ ( v T 1 B n UU T B T n v 1 ) 2 ] , β n := E [ Tr ( V T ⊥ B n UU T B T n V ⊥ ) 2 ] . Then, using Lemma A.2.9 we have,

Define a n := √ α n and b n := √ β n . Then using √ 1 + x ≤ 1 + x 2 , we have

The result then follows from Lemma A.2.5.

Lemma A.2.9. For all t > 0 , under subgaussianity (Definition 2.1), let U ∈ R d × m , α n := E [ ( v T 1 B n UU T B T n v 1 ) 2 ] , β n := E [ Tr ( V T ⊥ B n UU T B T n V ⊥ ) 2 ] and ηL 2 σ 2 λ 1 ≤ 1 4 then

where B n is defined in Eq 3.

Proof. Let A n := X n X T n and F n denote the filtration for observations i ∈ [ n ] . Then,

For T 1 ,

For T 2 ,

For T 3 ,

= 2 ( 3 L 2 σ 2 ) 3 λ 3 1 α n -1 +2 ( 3 L 2 σ 2 ) 3 × E   √ λ 1 λ 2 ∥ ∥ U T B T n -1 v 1 ∥ ∥ 2 2 Tr ( V T ⊥ B n -1 UU T B T n -1 V ⊥ ) √ 3 ηL 2 σ 2 √ 3 ηL 2 σ 2 λ 1 λ 2 Tr ( V T ⊥ B n -1 UU T B T n -1 V ⊥ )   ≤ 2 ( 3 L 2 σ 2 ) 3 λ 3 1 α n -1 + ( 3 L 2 σ 2 ) 3 E [ λ 1 λ 2 v T 1 B n -1 UU T B T n -1 v 1 Tr ( V T ⊥ B n -1 UU T B T n -1 V ⊥ ) 3 ηL 2 σ 2 ] + η ( 3 L 2 σ 2 ) 4 E [ λ 2 1 λ 2 2 Tr ( V T ⊥ B n -1 UU T B T n -1 V ⊥ ) 2 ] = 2 ( 3 L 2 σ 2 ) 3 λ 3 1 α n -1 + ( 3 L 2 σ 2 ) 2 λ 1 λ 2 η E [ v T 1 B n -1 UU T B T n -1 v 1 Tr ( V T ⊥ B n -1 UU T B T n -1 V ⊥ )] + η ( 3 L 2 σ 2 ) 4 λ 2 1 λ 2 2 E [ Tr ( V T ⊥ B n -1 UU T B T n -1 V ⊥ ) 2 ] ≤ 2 ( 3 L 2 σ 2 ) 3 λ 3 1 α n -1 + ( 3 L 2 σ 2 ) 2 λ 1 λ 2 η √ α n -1 β n -1 + η ( 3 L 2 σ 2 ) 4 λ 2 1 λ 2 2 β n -1 For T 4 , T 4 = E [ ( v T 1 A n B n -1 UU T B T n -1 A n v 1 ) 2 ] = E [ ( X T n v 1 v T 1 X n ) 2 ( X T n B n -1 UU T B T n -1 X n ) 2 ] = E [ E [ ( X T n v 1 v T 1 X n ) 2 ( X T n B n -1 UU T B T n -1 X n ) 2 ∣ ∣ ∣ ∣ F n -1 ]] ≤ ( 4 L 2 σ 2 ) 4 E [√ E [ ( v T 1 X n X T n v 1 ) 4 ∣ ∣ ∣ ∣ F n -1 ] E [ Tr ( B n -1 UU T B T n -1 Σ ) 4 ∣ ∣ ∣ ∣ F n -1 ] ] ≤ ( 4 L 2 σ 2 ) 4 E [√ ( v T 1 Σ v 1 ) 4 E [ Tr ( B n -1 UU T B T n -1 Σ ) 4 ∣ ∣ ∣ ∣ F n -1 ] ] , using Lemma A. 2 . 2 with p = 4 = ( 4 L 2 σ 2 ) 4 λ 2 1 E [√ ( Tr ( B n -1 UU T B T n -1 Σ )) 4 ] = ( 4 L 2 σ 2 ) 4 λ 2 1 E [ ( Tr ( B n -1 UU T B T n -1 V ⊥ Λ 2 V T ⊥ ) + λ 1 Tr ( B n -1 UU T B T n -1 v 1 v T 1 )) 2 ] ≤ 2 ( 4 L 2 σ 2 ) 4 λ 2 1 ( λ 2 2 β n -1 + λ 2 1 α n -1 ) Substituting in Eq A.31 along with using ηL 2 σ 2 λ 1 ≤ 1 4 we have, α n ≤ ( 1 + 4 ηλ 1 +100 η 2 L 4 σ 4 λ 2 1 ) α n -1 +100 η 2 L 4 σ 4 λ 2 1 √ α n -1 β n -1 +600 η 4 L 8 σ 8 λ 4 1 β n -1 Hence proved. Lemma A.2.10. For all t > 0 , under subgaussianity (Definition 2.1), let U ∈ R d × m , α n := E [ ( v T 1 B n UU T B T n v 1 ) 2 ] , β n := E [ Tr ( V T ⊥ B n UU T B T n V ⊥ ) 2 ] and ηL 2 σ 2 ≤ 1 4 min { 1 λ 1 , 1 Tr(Σ) , 1 √ λ 1 Tr(Σ) } then β n ≤ ( 1 + 4 ηλ 2 +100 η 2 log ( n ) L 4 σ 4 λ 2 Tr(Σ) ) β n -1 +100 η 2 log ( n ) L 2 σ 2 λ 1 Tr(Σ) √ α n -1 β n -1 +600 η 2 log 2 ( n ) L 4 σ 4 λ 2 1 α n -1 where B n is defined in Eq 3.

Then,

For T 1 ,

For T 2 ,

Hence proved.

Lemma A.2.11. Let U ∈ R d × m and u 0 ∼ N (0 , I d ) , then for all n ≥ 0 we have

Proof.

Lemma A.2.12. Let U ∈ R d × m and u 0 ∼ N (0 , I d ) , then for all n ≥ 0 we have

Proof. Let the eigendecomposition of B T n UU T B n for a fixed B n be given as P Λ P T such that PP T = P T P = I and Λ /followsequal 0 . Denote u 0 ≡ u and y := P T u . Therefore,

/negationslash

Note that E [ y ] = P T E [ u ] = 0 , E [ yy T ] = P T E [ uu T ] P = P T P = I . Therefore, y ∼ N (0 , I d ) . Therefore, E [ y 4 i ] = 3 and E [ y 2 i y 2 j ] = E [ y 2 i ] E [ y 2 j ] = 1 . Therefore,

/negationslash

Substituting in Eq A.33, we have

/negationslash

A.3 Proofs of entrywise deviation of Oja's vector

We first state some useful results here. Let a 0 = v 1 ( i ) 2 , b 0 = Tr ( V T ⊥ e i e T i V ⊥ ) = 1 -v 1 ( i ) 2 . Let the learning rate, η , be set according to Lemma A.2.4. Note that (1 + x ) ≤ exp( x ) , ∀ x ∈ R . From Lemma A.2.7, we have

Similarly, from Lemma A.2.8 and using ( a + b ) 2 ≤ 2 a 2 +2 b 2 , we have

Finally, noting that (1 + x ) ≥ exp ( x -x 2 ) ∀ x ≥ 0 , we have

Now we are ready to provide proofs of Lemmas 3.11 and 3.12.

Lemma 3.11 (Tail bound in support) . Fix a δ ∈ (0 . 1 , 1) . Define the event G := { | v T 1 u 0 |≥ δ √ e } and threshold τ n := δ √ 2 e min i ∈ S | v 1 ( i ) | (1 + ηλ 1 ) n . Let the learning rate be set as in Lemma 3.1. Then, for an absolute constant C H > 0 ,

Proof of Lemma 3.11. Let u 0 = av 1 + bv ⊥ for v T ⊥ v 1 = 0 and a = u T 0 v 1 , and b = u T 0 v ⊥ ∈ R for some vector v ⊥ orthogonal to v 1 . Then ∀ i ∈ S ,

Then,

For convenience of notation, define γ n := √ e δ τ n and q i := | e T i B n v 1 | . Then,

We now bound T i using Eq A.35 and Eq A.39 as -

The second inequality follows from the fact that 1 + x ≥ exp( x -x 2 ) , for x ≥ 0 . Note that for x ∈ (0 , 1) , exp x ≤ 1 + 2 x . Therefore, for ( 8 L 4 σ 4 +2 ) η 2 nλ 2 1 ≤ 1 4 , we have

The result then follows by using Claim(2) in Lemma A.2.4.

We next provide the proof of Lemma 3.12.

Lemma 3.12 (Tail bound outside support) . Fix a δ ∈ (0 . 1 , 1) . Let the learning rate be set as in Lemma 3.1 and define the threshold τ n := δ √ 2 e min i ∈ S | v 1 ( i ) | (1 + ηλ 1 ) n . Then, for min i | v 1 ( i ) | = ˜ Ω ( λ 1 λ 1 -λ 2 ( d n 2 ) 1 4 ) and an absolute constant C T > 0 we have,

Proof of Lemma 3.12. Note that for i / ∈ S , a 0 = 0 , b 0 = 1 . Therefore, we have,

Define

Note that using Assumptions 2,

where the last inequality follows for sufficiently large n mentioned in the theorem statement. Therefore, using Eq A.46 and A.47 along with Claim (3) from Lemma A.2.4, g i is bounded as

where the last inequality used Claim (4) from Lemma A.2.4. Therefore, we have,

We now bound R i . Therefore,

where (i) uses Lemmas A.2.11 and A.2.12. Denote the numerator and denominator of R i as N ( R i ) and D ( R i ) . For the numerator N ( R i ) using Eq A.44 and A.45, we have

where the last inequality follows from Claim (4) in Lemma A.2.4. For the denominator D ( R i ) , using Eq A.50

Recall that for x ∈ (0 , 1) , exp( x ) ≤ 1 + 2 x . Therefore, for ( 100 L 4 σ 4 +2 ) η 2 nλ 2 1 ≤ 1 4 which holds due to Claim (2) from Lemma A.2.4, substituting in Eq A.51, we have

A.4 Proof of convergence for support recovery (Lemma 3.1,Theorem 3.2)

We start with the proof of Lemma 3.1.

Proof. Let E := { S ⊆ ̂ S } and set δ := 1 50 for this proof. We upper bound P ( E c ) . Define r i := e T i B n u 0 , i ∈ [ d ] . Observe that

or equivalently,

Therefore, for any fixed τ > 0

Let G := { | v T 1 u 0 |≥ δ √ e } and threshold τ n := δ √ 2 e min i ∈ S | v 1 ( i ) | (1 + ηλ 1 ) n . Using a unionbound,

Using Lemma A.2.1, we have P ( G c ) ≤ δ . We bound T 1 and T 2 using Lemmas 3.11 and 3.12 respectively. Therefore,

where the last inequality follows by using the bound on n .

Next, using Lemma 3.1, we prove Theorem 3.2.

Theorem 3.2 (High probability support recovery) . Let Assumptions 1, 2 hold. For dataset D := { X i } i ∈ [ n ] , let A be the randomized algorithm which computes ̂ S ← OjaSupportRecovery ( { X i } i ∈ [ n ] , k, η ) , where η := 3 log( n ) n ( λ 1 -λ 2 ) and k = s . Then, for δ ∈ (0 , 1) min i | v 1 ( i ) | = ˜ Ω ( λ 1 λ 1 -λ 2 ( d n 2 ) 1 4 ) , ˜ S ← SuccessBoost ( { X i } i ∈ [ n ] , A , δ ) satisfies, P ( ˜ S = S ) ≥ 1 -δ

,

Proof. Consider the set T := {S : S ⊆ [ d ] , |S| = s } with the associated metric ρ ( S , S ′ ) := ( S /negationslash = S ′ ) .

✶ Then, Lemma 3.1 shows that the randomized algorithm, A , is a ConstantSuccessOracle ( D , θ, T , ρ, S, 0) (Definition 3.9).

Therefore, the result follows from Lemma 3.10.

A.5 Proof of convergence for sparse PCA (Theorems 3.5,3.7)

We start by providing the proof of Theorem 3.5 in Section A.5.1, and then provide the proof of Theorem 3.7 in Section A.5.2.

A.5.1 Proof of theorem 3.5

Let ̂ v := B n w 0 ‖ B n w 0 ‖ 2 . Then, for any subset ˆ S ⊆ S (obtained from a support recovery procedure such as Algorithm 1), the corresponding output of a truncation procedure with respect to ̂ S is given as:

We first prove a general and flexible result that bounds the sin 2 error as a function of B n (see Eq 3) and analyze the performance of v trunc by viewing it as a power method on w 0 followed by a truncation using the set ̂ S in the following result.

Lemma A.5.1. Let B n and v trunc be defined as in Eq 3 and Eq A.53 respectively. For ̂ S ⊆ [ d ] such that ̂ S ⊥ ⊥ B n , w 0 , then with probability at least 1 -δ

where C is an absolute constant and δ ∈ (0 , 1) .

Proof. Using the definition of sin 2 error,

For the denominator, with probability at least (1 -δ ) , we have

where ( i ) follows from Lemma A.2.1. For the numerator, using ζ 2 from Lemma 3.1 of [JJK + 16], with probability at least (1 -δ ) , we have

Combining Eq A.55 and Eq A.56 with Eq A.54 completes our proof.

Lemma A.5.1 provides an intuitive sketch of our proof strategy. Following the recipe proposed in [JJK + 16], we show how to upper-bound /epsilon1 n := C log ( 1 δ ) δ 2 Tr ( B T n ( I ̂ S -I ̂ S v 1 v T 1 I ̂ S ) B n ) v T 1 B T n I S ∩ ̂ S B n v 1 . For upperbounding the numerator, we bound E [ Tr ( B T n ( I ̂ S -I ̂ S v 1 v T 1 I ̂ S ) B n )] and use Markov's inequality. To lower-bound the denominator, we lower-bound E [ v T 1 B T n I S ∩ ̂ S B n v 1 ] , upper-bound the variance E [ ( v T 1 B T n I S ∩ ̂ S B n v 1 ) 2 ] and finally use Chebyshev's inequality. A formal analysis is provided in the following theorem -

Theorem A.5.2 (Convergence of Truncated Oja's Algorithm) . Let ̂ S ⊆ [ d ] be the estimated support set, such that ̂ S ⊥ ⊥ B n , w 0 (see Algorithm 2). Consider any event E solely dependent on the randomness of ̂ S . Define:

Fix δ ∈ (0 . 1 , 1) . Set the learning rate as η := 3 log( n ) n ( λ 1 -λ 2 ) . Then, under Assumption 2, for n = Ω ( s ( λ 1 log( n ) λ 1 -λ 2 ) 2 ) and p 0 ( 1 + δ 16 ) ≤ 1 + 2 ηλ 1 s ( 4 λ 1 ( λ 1 -λ 2 ) ) , we have with probability at least 1 -δ -P ( E c ) ,

where v trunc is defined in Eq A.53 and C ′ > 0 is an absolute constant.

Proof of Theorem A.5.2. We first note that from Lemma A.5.1, with probability at least 1 -δ ,

Next, we bound χ , conditioned on the event E . Using Markov's inequality, we have with probability at least 1 -δ ,

Note that (1 + x ) ≤ exp( x ) ∀ x ∈ R . From Lemma A.2.7, we have

where the last inequality follows due to Lemma A.2.4.

Substituting Eq A.59 and Eq A.60 in Eq A.58, we have with probability at least (1 -δ ) , conditioned on the event E ,

Similarly, for the denominator we have with probability at least 1 -δ using Chebyshev's inequality, conditioned on the event E ,

Recall that p 0 := v T 1 E [ I S ⋂ ̂ S ∣ ∣ ∣ ∣ E ] v 1 . Using the argument from Lemma 11 from [JJK + 16] and ̂ S ⊥ ⊥ B n ,

This is since the base case of their recursion, [JJK + 16] has v T 1 Iv 1 which is 1, but we have v T 1 E [ I S ⋂ ̂ S ∣ ∣ ∣ ∣ E ] v 1 which is defined as p 0 .

Next, using Lemma A.2.8 and noting that v T 1 I S ⋂ ̂ S v 1 +Tr ( V T ⊥ I S ⋂ ̂ S V ⊥ ) = Tr ( I S ⋂ ̂ S ) we have,

where in the last inequality, we used ( v T 1 I S ⋂ ̂ S v 1 ) 2 ≤ v T 1 I S ⋂ ̂ S v 1 ≤ 1 . For convenience of notation, we define

where we used ηλ 1 s ( 4 λ 1 ( λ 1 -λ 2 ) ) ≤ 1 2 (due to Lemma A.2.4) and p 0 ≤ 1 . Substituting Eq A.63 and Eq A.64 in Eq A.62, and we have with probability at least (1 -δ ) ,

conditioned on E ,

v

T

1

B

n

G

S

B

̂

T

n

v

1

where in ( i ) we used ( 100 L 4 σ 4 +4 ) η 2 nλ 2 1 ≤ 1 and x ∈ (0 , 1) , exp( x ) ≤ 1 + 2 x . For ( ii ) , it suffices to have

which is further ensured by,

Note that for the choice of η , and δ ≥ 1 10 , we have using Lemma A.2.4,

for sufficiently large n . Therefore, we only ensure that p 0 is greater than the first term in the theorem statement. Finally, let

Using Eq A.61 and Eq A.65 and substituting in Eq A.57, we have with probability at least 1 -2 δ , conditioned on E , χ ≤ ξ , or equivalently P ( χ ≥ ξ ∣ ∣ ∣ ∣ E ) ≤ 2 δ . Therefore,

The proof follows by making δ smaller by a constant factor.

With Theorem A.5.2 in place, we are ready to finally provide the proof of one of our main results, Theorem 3.5.

Theorem 3.5 (Vector Truncation) . Let Assumptions 1 and 2 hold and k ≥ s . For dataset D := { X i } i ∈ [ n ] and w 0 ∼ N (0 , I ) , let A be the randomized algorithm which computes ˆ v truncvec ← TruncateOja ( { X i } i ∈ ( n 2 ,n ] , ̂ S, Oja , { η, w 0 } ) , where η := 3 log( n ) n ( λ 1 -λ 2 ) . Then, for min i | v 1 ( i ) | = ˜ Ω ( ( d n 2 ) 1 8 ) , ˜ v ← SuccessBoost ( { X i } i ∈ [ n ] , A , d -10 ) satisfies,

with probability at least 1 -d -10 , where C ′′ ≥ 0 is an absolute constant.

Proof. Let E := { S hi ⊆ ̂ S } and set δ := 1 4 for this proof. Consider the following variables from Theorem A.5.2:

Since | ̂ S | = k , therefore,

Furthermore, under event E , S hi ⊆ { S ⋂ ̂ S } . Therefore,

To verify the assumption on p 0 mentioned in Theorem A.5.2, it is sufficient to ensure

which is true by the definition of η and n (see Lemma A.2.4). Lastly,

Therefore, using bounds on β 0 , p 0 and α 0 from Eqs A.66, A.67 and A.68 respectively, in conjunction with Theorem A.5.2, with probability at least 1 -δ -P ( E c ) ,

Using Lemma 3.1, P ( E c ) ≤ 5 δ . The result then follows using Eq A.69 and setting δ smaller by a constant.

A.5.2 Proof of theorem 3.7

We first state the result from [Lia23] achieving the optimal sin 2 error rate for Oja's Algorithm.

Proposition A.5.3 (Optimal Rate for Oja's Algorithm with Subgaussian Data (Theorem 3.1, [Lia23])) . Let { X i } i ∈ [ n ] be i.i.d samples from a subgaussian distribution (Definition 2.1) with covariance matrix, Σ , leading eigenvector v 1 and eigengap, λ 1 -λ 2 > 0 . Then, there exists an algorithm OptimalOja which operates in O ( d ) space, O ( nd ) time, processes one datapoint at a time, and returns an estimate ˆ v which satisfies, with probability at least 1 -δ , δ ∈ (0 , 1)

where C > 0 is an absolute constant.

Theorem 3.7 (Data Truncation) . Let Assumptions 1 and 2 hold and k ≥ s . For dataset D := { X i } i ∈ [ n ] and w 0 ∼ N (0 , I ) , let A be the randomized algorithm which computes ˆ v truncvec ←

with probability at least 1 -d -10 , where C ′′ ≥ 0 is an absolute constant.

Proof. Let δ := 1 3 . Define the event E = { S ⊆ ̂ S } . Using Lemma 3.1, we have that P ( E ) ≥ 1 -δ

Let χ := sin 2 (ˆ v truncvec , v 1 ) and ξ := C λ 1 λ 2 ( λ 1 -λ 2 ) 2 k log ( 1 δ ) n for an absolute constant C > 0 . Therefore,

Therefore, next we bound P ( sin 2 (ˆ v truncvec , v 1 ) ∣ ∣ ∣ ∣ E ) . Therefore, we seek to bound the sin 2 error after truncating the data using the true support, S . Note that

Therefore, after truncation, the leading eigenvector and eigenvalue are preserved, and the second largest eigenvalue is at most λ 2 . Furthermore, the truncated distribution is still subgaussian, and therefore Proposition A.5.3 is applicable here and we have with probability at least 1 -δ ,

Eq (A.70) and (A.71) show that A is a ConstantSuccessOracle ( D , ( η, k ) , T , ρ, v 1 , O ( k log ( d ) /n )) (Definition 3.9) for the set T = { u : u ∈ R d , ‖ u ‖ 2 = 1 } with the metric ρ ( u, v ) := ∥ ∥ uu /latticetop -vv /latticetop ∥ ∥ F = 1 2 | sin ( u, v ) | . The result then follows from Lemma 3.10.

A.6 Alternate method for truncation

In this section, we present another algorithm for truncation, based on a value-based thresholding, complementary to the technique described in Section 3. The proof technique uses the same tools as the ones described in Section 3. Both Algorithm 2 and 4 may be of independent interest depending on the particular use-case and constraints of the particular problem. Theorem A.6.1 provides the convergence guarantees for Algorithm 4. Note that compared to Theorem 3.5, Theorem A.6.1 provides a better guarantee for the sample size. However, this comes at the cost of the sparsity of the returned vector, ˆ v oja-thresh , not being a controllable parameter. We can however show that the support size of ˆ v oja-thresh is O ( s ) in expectation. For the purpose of this proof, let

Theorem A.6.1 (Convergence of Oja -Thresholded ) . Let ˆ v oja-thresh , ̂ S be obtained from Algorithm 4. Set the learning rate as η := 3 log( n ) n ( λ 1 -λ 2 ) . Define threshold γ n := 3 4 √ 2 e min i ∈ S hi | v 1 ( i ) | (1 + ηλ 1 ) n . Then for n = ˜ Ω ( 1 s min i ∈ S hi v 1 ( i ) 4 ( λ 1 λ 1 -λ 2 ) 2 ) , we have E [ | ̂ S | ] ≤ C ′ s and with probability at least 3 4 , sin 2 ( ˆ v oja-thresh , v 1 ) ≤ C ′′ ( λ 1 λ 1 -λ 2 ) 2 max { s, log ( d ) } log ( d ) n , S hi ⊆ ̂ S where C ′ , C ′′ > 0 are absolute constants.

1: Input : Dataset { X i } i ∈ [ n ] , learning rate η > 0 , truncation threshold γ n

2: Set b n ← 0 and choose y 0 , w 0 ∼ N (0 , I ) independently

4: y t ← ( I + ηX t X T t ) y t -1

3: for t in range[1, n 2 ] do

5: b n ← b n +log( ‖ y t ‖ 2 )

6: y t ← y t ‖ y t ‖ 2

7: end for

8: ̂ S ← Set of indices, i ∈ [ d ] , such that log ( | e T i y n | ) + b n -log ( γ n ) ≥ 0 .

10: ˆ v oja-thresh ← /floorleft ̂ v /floorright ̂ S ∥ ∥ /floorleft ̂ v /floorright S ∥ ∥

̂ 2 11: return [ ˆ v oja-thresh , ̂ S ]

Proof. Consider the setting of Theorem A.5.2. Set δ := 1 4 for this proof and let E be the event { | v T 1 y 0 |≥ δ √ e } . By Lemma A.2.1, P ( E ) ≥ 1 -δ . Recall the definitions,

We upper bound α 0 , β 0 and lower bound p 0 under the setting of Algorithm 4. Define r i := e T i B n y 0 , i ∈ [ d ] . For α 0 , p 0 , we have

For β 0 we have

The result then follows using Theorem A.5.2 and substituting the bounds on α 0 , β 0 and p 0 . Finally, note that using a similar argument as Theorem 3.5, we have

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We present a O ( d ) space and O ( nd ) time algorithm for Sparse PCA for general covariance matrices.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Section 3.2 shows that we handle general covariance matrices in nearly linear time, albeit at a worse sample size.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate 'Limitations' section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All Proofs are in the Appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Figure 1a and Figure 2 contains the experimental setup in the caption.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We make the code for our experiments available in the supplementary material. Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Figure 1a and Figure 2 contains the experimental setup in the caption.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Our experiments were performed on a single Macbook Pro M2 2022 CPU with 8 GB RAM.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer 'Yes' if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Figure 1a provides error bars over 100 random runs. Figure 2 plots average over 10 runs.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We abide by the NeurIPS Code of Ethics in our work.

Guidelines:

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper does not release any models

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: There are no datasets or existing codebases used.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets to require documentation or licensing. Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We only use simulated data for our experiments.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: We do not have experiments with crowdsourcing or human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper studies the problem of finding the top eigenvector from samples.
Given $n$ samples drawn from a distribution whose mean is $0$ and covariance matrix is $\Sigma\in\mathbb{R}^{d\times d}$, we would like to find a vector $\hat{v}$ based on these $n$ samples such that $\hat{v}$ and the top eigenvector of $\Sigma$, $v$, have a small $\sin^2$ error which is defined as $1-\langle\hat{v},v\rangle^2$.
Furthermore, we assume that the sparsity of $v$ is $s$ and hence we expect the output $\hat{v}$ also has sparsity $s$.
The goal in this paper is to design a single pass algorithm using $O(nd)$ time and $O(d)$ space such that the error is minimized.
Oja's algorithm is a well known algorithm for finding the top eigenvector and works as follows.
In the $t$-step, we iteratively update our current solution towards the $t$-th sample.
The authors applied Oja's algorithm to achieve the goal and showed that the error is at most $O(\frac{\sigma_*^2 s\log d}{n})$ where $\sigma_*^2$ depends on the top two eigenvalues.

STRENGTHS:
- The problem seems to be a natural question and well-motivated.

- The general presentation is good.
The readers of all levels of expertise should be able to follow the main idea in this paper.

WEAKNESSES:
- In terms of techniques, the main idea is to apply the known Oja' algorithm.
I am not sure if there are any fundamental new ideas introduced in this paper.

QUESTIONS:
Note:

- Line 31: $r_{\textsf{eff}}$ is not defined yet in the main text.

LIMITATIONS:
.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-07

comments: SUMMARY:
The work proposes a one pass Ojas' algorithm which can achieve minimax error bound for high dimensional sparse PCA under standard technical conditions.

STRENGTHS:
The paper is extremely well written. The proposed one-pass Oja's algorithm is novel with detailed convergence analysis and convincing numerical experiments to support author's claims. I appreciate the thorough literature review highlighting how the current paper improves on over the previous works (in particular Table 1). The relaxation on effective rank assumption is particularly notable in the results. The mathematical ideas in the proofs are easy to follow and contain several new techniques.

WEAKNESSES:
In general, I like the paper as it is quite clear and mathematically sound. In some minor places, the exposition can be slightly improved by defining terminologies before using them (for instance please define $\sin^2$ error before referring).

QUESTIONS:
Q: Can something be said along the lines of these results for other top eigenvectors of $\Sigma$ (not just $v_1$)?

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the problem of streaming sparse PCA under iid data. That is, we have $x_1,...,x_n \\sim \\mathcal D$ iid vectors in $\\mathbb R^d$ which are revealed to us in an online fashion. We want to estimate the top eigenvector of $\\Sigma = \\mathbb E[xx^\\intercal]$. We assume that this top eigenvector $v_1$ is $s$-sparse, for $s = O(\\frac{n}{\\log(n)})$. The error between the real and estimated top eigenvector is computed using the $\\sin^2$ error metric $(1-\\langle v_1, \\tilde v_1\\rangle^2)$. \n\nThis paper shows that Oja's algorithm can be used to both find the support of $v_1$ and the values in $v_1$ (assuming we are given some value $k \\geq s$ and correctly choose a step size parameter).\nThis algorithm runs in $O(nd)$ time and $O(d)$ space.\nThe key technical assumptions made are:\n- The data is iid subgaussian\n- The sparsity of $v_1$ is $s = O(\\frac{n}{\\log (n)})$\n- The effective dimension (ratio of trace to spectral norm) of the covariance matrix is at most $O(\\frac{n}{\\log(n) \\sigma^2})$ where $\\sigma^2 = \\frac{\\lambda_1}{\\lambda_1 - \\lambda_2} \\cdot \\frac{\\lambda_2}{\\lambda_1 - \\lambda_2}$ measures the (square) of the singular value gap between the top two eigvals of $\\Sigma$.\n- The smallest nonzero entries of the (unit vector) top eigevnector $v_1$ is $\\tilde\\Omega(\\frac{d^{1/8}}{n^{1/4}})$.\n\nIn contrast to prior works, this work achieves better $\\sin^2$ error that prior $O(nd)$ time algorithms and $O(d)$ space algorithms. It also makes no assumption about the quality of the starting vector (i.e. it's not a local convergence result, it's a global convergence result).\n\nThe results are essentially all theoretical, with one experiments used to show that Oja works well here, and another used to elucidate which theoretical bounds are loose.\n\nSTRENGTHS:\nThe paper is a nice contribution to the literature, and it's well written. Sparse PCA is an important problem, and handling it in low space and time is important as well. The error metrics and assumptions are all reasonable. The paper is written clearly. It's just sorta all around solid.\n\nThe result is original in that it has a clear goal: achieve the $\\sin^2$ error of large-space or large-time algorithms (those that use $\\omega(nd)$ time or $\\omega(d)$ space), but only using $O(nd)$ time and $O(d)$ space. It's especially nice that a pretty naive application of Oja's algorithm can achieve this.\n\nThere's also some nice novelty in the proof technique, which bounds the 2nd and 4th moments of the entries of the output of Oja's algorithm. In particular, the authors tighten a bound from the prior work by designing and solving a system of linear recurrences. A cool math setup that I do not often see.\n\nI'm not an expert in the sparse pca world, not the streaming pca world, and certainly not streaming sparse pca. That said, assuming the authors are not omitting any relevant prior works, this results on low error in very low space and time seems pretty cool. It seems like a particularly nice step in the sparse PCA literature.\n\nWEAKNESSES:\nThere's a few notation inconsistency issues, some minor gripes. Nothing I'm really worried about. I'll push it all to the \"questions\" section below.\n\nI accept this paper for publication.\n\nQUESTIONS:\nNone of these are game-breaking, and many of these are minor typos. Feel free to ignore whatever feels unfair to you. But, do at least make the notation for the initial vector consistent and make the figures easy to read.\n\n1. Annoyingly, the authors __very often__ swap the symbols $y_0$, $u_0$, $w_0$, and $z_0$. Please fix this.\n1. Figure 1 is too hard to read. The letters are too small. The axes have no labels. The error is negative somehow (if it's plotting $\\pm1$ standard deviation, which is making the error bars negative, consider using 25th and 75th quantiles instead?).\n1. It's not clear if Lemma 3.1 and Theorem 3.2 allow us to use over/underestimates of $\\eta$, or if the theorems are very tied to that exact value of $eta$. Seems worth discussing.\n1. It's not clear why Theorem 3.2 requires $k=s$ instead of $k\\geq s$. Discussing this would be nice. Returning a vector whose support is $\\log(1/\\delta)$ times larger than $s$ seems fine to me, if that's the issue here.\n1. Prop 3.4 is written a bit unclearly, namely around the \"with sparsity parameter, $n=$\" part, since $n$ is not a sparsity parameter.\n1. Theorems 3.5 and 3.7 uses failure probability $d^{-10}$, which is fine I guess? But why not just use a $\\delta$?\n1. Remark 3.6 seems like it might make more sense to have back around assumption 2\n1. Section 3.4 strikes me as a very standard analysis style. Not sure why you point to [KLL+23] specifically. You can say it's standard, and point to [KLL+23] as an example of this, maybe? If I'm missing something and it's not standard, lemme know.\n1. Figure 2 is also too hard to read, especially in print. The letters are too small. The series look too much like each-other. The y-axis should be more specific in the error. It's not really clear what the dotted / population lines are showing -- is it $\\log(E[e_i^\\intercal B_n u_0])$, or is it $E[\\log(e_i^\\intercal B_n u_0)]$? Is it something else? What exactly does \"error\" mean here?\n1. Line 221 can use $E[r_i | u_0]$ imo\n1. Idk why equation (4) uses absolute values and $\\pm$ on the non-top-eigenvalue terms. Isn't $E[B_n]$ PSD, and thus the second term guaranteed to be nonnegative?\n1. Line 221 maybe mention that $E[B_n] = (1+\\eta \\Sigma)^t^?\n1. Line 232 this line about [SSM11] taking $\\lambda_1 = d^\\alpha \\rightarrow \\infty$ is kinda confusing because it's not a scale-invariant claim. Is it like a condition number that's getting large?\n1. Line 238 should mention that Section 4 explains the technique a bit more in detail\n1. Line 247 should really point to Section 4 as well, explaining the technique in a bit more detail\n1. [Line 256] If you have space, it'd be nice to understand why Theorem 3.5 needs a more general argument that $U = e_i$.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-15"}

comments: SUMMARY:
The paper studies Principal Component Analysis with O(d) space and O(nd) time, where n is the number of datapoints and d is their dimensionality.

The authors provide the first single-pass algorithm that under a general \Sigma matrix whose top principal vector is s-sparse, manages to find a close enough vector in the sense of the sinus-squared error. Their main theorem is Th. 1.1. which states that under a structural assumption on the effective rank (ratio of the trace and the principal eigenvalue of the population covariance matrix Σ) not being too large (the function involves the spectral gap and the top eigenvalue).

Their algorithm relies on Oja's algorithm and the authors show that w.h.p. the Oja vector when initialized by a random unit vector, will actually converge to an output whose top k entries in terms of magnitude will include the true support of the s-sparse v_0. Then, the authors can ue the recovered support and achieve minimiza optimal sparse PCA, thus improving upon several prior works.

STRENGTHS:
+very interesting and well-motivated problem

+clean framework and clean algorithm

+novel analysis and simple algorithm that improves upon several prior works

WEAKNESSES:
-no serious weaknesses

QUESTIONS:
Some typos: 
-matrix multiplication constant is not 2.732 as written.

LIMITATIONS:
-
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-18