Bootstrapping the Error of Oja's Algorithm 

1 Introduction . Since its discovery over a century ago [ 13 ] , principal component analysis ( PCA ) has been a cornerstone of data analysis . In many applications , dimension reduction is paramount and PCA offers an optimal low-rank approximation of the original data . PCA is also highly interpretable as it projects the dataset onto the directions that capture the most variance known as principal components . Important applications of PCA include image and document analysis , where the largest few principal components may be used to compress a large dimensional dataset to a manageable size without incurring much loss ; for a discussion of some other applications of PCA , see for example , [ 28 ] . In these settings , the original dimensionality , which could be the number of pixels in an image or the vocabulary size after removing stop-words , is in the tens of thousands . An offline computation of the principal components would require the computation of eigenvectors of the sample covariance matrix . However , in high-dimensional settings , storing the covariance matrix and subsequent eigen-analysis can be challenging . Streaming PCA methods have gained significant traction owing to their ability to iteratively update the principal components by considering one data-point at a time . One of the most widely used algorithms for streaming PCA is Oja ’ s algorithm , proposed in the seminal work of [ 41 ] . Oja ’ s algorithm involves the following update rule : wt+1 − wt = η ( wTt Xt ) Xt ; wTt+1wt+1 = 1 , ( 1 ) where Xt ∈ Rd is the tth data point and wt is the current estimate for the leading eigenvector of Σ = EXXT after t data-points have been seen . The parameter η can be thought of as a learning rate , which can either be fixed or varied as a function of t. In this paper we fix the learning rate , similar to [ 26 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Contribution : In the present work , we consider the problem of uncertainty quantification for the estimation error of the leading eigenvector from Oja ’ s algorithm , which is one of the most commonly used streaming PCA algorithms . Our contributions may be summarized as follows : 1 . We derive a high-dimensional weighted χ2 approximation to the sin2 error for the leading eigenvector of Oja ’ s algorithm . We recover the optimal convergence rate O ( 1/n ) while allowing d to grow at a sub-exponential rate under suitable structural assumptions on the covariance matrix , matching state-of-the-art theoretical results for consistency of Oja ’ s algorithm . Our result provides a distributional characterization of the sin2 error for Oja ’ s algorithm for the first time in the literature . The approximation holds for a wide range of step sizes . 2 . Since the weighted χ2 approximation depends on unknown parameters , we propose an online bootstrap algorithm and establish conditions under which the bootstrap is consistent . Our bootstrap procedure allows the approximation of important quantities such as the quantiles of the error associated with Oja ’ s algorithm for the first time . Prior analysis of Oja ’ s algorithm . While Oja ’ s algorithm was invented in 1982 it was not until recently that the theoretical workings of Oja ’ s algorithm have been understood . A number of papers in recent years have focused on proving guarantees of convergence of the iterative update in ( 1 ) toward the principal eigenvector of the ( unknown ) covariance matrix EXXT , which can be recast as stochastic gradient descent ( SGD ) on the quadratic objective function min w wTw=1 −trace ( wTΣw ) , Σ = EXXT , ( 2 ) projected onto the non-convex unit sphere . We assume that the data-points are mean zero . Despite being non-convex and thus falling outside the framework for which theory for stochastic gradient descent convergence is firmly established , the output of Oja ’ s algorithm be viewed as a product of random matrices and shares similar structure to other important classes of non-convex problems , such as matrix completion [ 27 , 29 ] , matrix sensing [ 27 ] , and subspace tracking [ 4 ] . Thus , studying this optimization problem serves as a natural first step toward understanding the behavior of SGD in more general non-convex settings . Let v1 denote the principal eigenvector of Σ , and let v̂1 = wn be the solution to the stochastic iterative method applying Eq 1 . Finally , let λ1 > λ2 be the first and second principal eigenvalues of Σ . Sharp rates of convergence for Oja ’ s updates were established in [ 25 ] . Under boundedness assumptions on ‖XiXTi − Σ‖ , they show that with constant probability , the square of the sine of the angle between v1 and w satisfies : 1− ( vT1 v̂1 ) 2 = O ( 1 n ) ( 3 ) where the O hides a constant which depends in the optimal way on the eigengap between the top two eigenvalues , and independent of n or d , improving on previous error bounds for Oja ’ s algorithm [ 46 , 18 , 3 , 47 , 38 , 2 ] which showed convergence rates that deteriorate with the ambient dimension d , and thus did not fully explain the efficiency of Oja ’ s update . This sharp rate is remarkable , as it matches the error of the principal eigenvector of the sample covariance matrix , which is the batch or offline version of PCA . Other notable work include [ 31 , 33 ] for unbounded Xi , analysis of Oja ’ s algorithm for computing top k principal components [ 1 , 24 ] . The bootstrap . The bootstrap , proposed by [ 9 ] , is one of the most widely used methods for uncertainty quantification in machine learning and statistics and accordingly has a vast literature . We refer the reader to [ 17 , 49 ] for expositions on the classical theory of the bootstrap for IID data . Recently , since the groundbreaking work of [ 7 , 8 ] , the bootstrap has seen a renewed surge of interest in the context of high-dimensional data where d can be potentially exponentially larger than n. Of particular relevance to the present work are high-dimensional central limit theorems ( CLTs ) for quadratic forms , which have been studied by [ 43 , 51 , 15 ] . In particular , our CLT for the estimation error of Oja ’ s algorithm invokes a modest adaptation of [ 51 ] to independent but non-identically distributed random variables . In machine learning , bootstrap methods have been used to estimate the uncertainty of randomized algorithms such as bagging and random forests [ 35 ] , sketching for large scale singular value decomposition ( SVD ) [ 36 ] , randomized matrix multiplication [ 37 ] , and randomized least squares [ 34 ] . A standard notion of bootstrap consistency is that , conditioned on the data , the distribution of the suitably centered and scaled bootstrap functional approaches the true distribution with high probability in some norm on probability measures , typically the Kolmogorov distance , which is the supremum of the absolute pointwise difference between two CDFs . Bootstrap consistency is often established by deriving a Gaussian approximation for the sampling distribution and showing that the bootstrap distribution is close to the corresponding Gaussian approximation with high probability . It may seem that if one knows that the approximating distribution of a statistic is Gaussian , this defeats the purpose of bootstrap . However , for most statistics , the parameters of the normal approximation depend on unknown model parameters , and have to be estimated if one intends to use the normal approximation . Furthermore , the CLT only gives a first-order correct approximation of the target distribution , i.e . with O ( 1/ √ n ) error . In contrast , the bootstrap of a suitably centered and scaled statistic has been shown to be higher order correct for many functionals [ 16 , 17 , 19 ] . Quantifying uncertainty for SGD . Behind the recent success of neural networks in a wide range of sub-fields of machine learning , the workhorse algorithm has become Stochastic gradient descent ( SGD ) [ 42 , 40 , 44 ] . For establishing consistency of bootstrap , one requires to establish asymptotic normality [ 11 , 42 , 45 , 39 ] . There has also been many works on uncertainty estimation of SGD [ 6 , 32 , 12 , 48 ] . However , all these works are for convex , and predominantly strongly convex loss functions . Only recently , [ 52 ] has established asymptotic normality for nonconvex loss functions under dissipativity conditions and appropriate growth conditions on the gradient , which are weaker conditions than strong convexity but not significantly so . Now , in Section 2 we present notation and do setup , present our main theoretical results in Section 3 , followed by simulations in Section 4 . 2 Preliminaries . We consider a row-wise IID triangular array , where the random vectors { Xi } in the nth row take values in Rdn , with E [ Xi ] = 0 and Var ( Xi ) = Σn . Note that the triangular array allows { X1 , . . . , Xn } come from a different distribution for each n and the setting where d is fixed and n grows is a special case . For readability , we drop the subscript n from Σn . We use ‖ · ‖ to denote the Euclidean norm for vectors and the operator norm for matrices and ‖ · ‖F to denote the Frobenius norm . Expanding out the recursive definition in Eq 1 , we see that Oja ’ s iteration can be expressed as wt+1 = ( Id + ηXtX T t ) wt . Thus , after n iterations the vector can be written as a matrix-vector product , where the matrix is a product of n independent matrices . Expanding out the recursive definition , we get : Bn : = n∏ i=1 ( Id + ηXiX T i ) v̂1 = Bnu0 ‖Bnu0‖ , ( 4 ) where Id is a d× d identity matrix . where u0 is a random unit vector in d dimensions . In the scalar case , when η = 1/n , for large n , the numerator of Eq 4 behaves like exp ( ∑ iX 2 i /n ) , which in turn converges to exp ( E [ X21 ] ) . For matrices , one hopes that , by independence , a result of the same flavor will hold . And in fact if it does hold , then for η = lognn , the numerator in Eq 4 will concentrate around exp ( log nΣ ) . The spectrum of this matrix is dominated by the principal eigenvector , i.e . the ratio of the first eigenvalue to the second one is exp ( log n ( λ1 − λ2 ) ) , where λi is the ith eigenvalue of the covariance matrix Σ . This makes it clear that Oja ’ s algorithm is essentially a matrix vector product of this matrix exponential ( suitably scaled ) and a random unit vector . However , the intuition from the scalar case is nontrivial to generalize to matrices due to noncommutativity . Limits of products of random matrices have been studied in mathematics in the context of ergodic theory on Markov chains ( see [ 14 , 30 , 5 , 10 ] etc. ) . However , until recent results of [ 23 ] , which extended and improved results in [ 21 ] , there has not been much work on quantifying the exact rate of convergence , or finite-sample large deviation bounds for how a random matrix product deviates from its expectation . We reparametrize η as ηn/n , where ηn is chosen carefully to obtain a suitable error rate . Note that this is not a scheme where we decrease η over time as in [ 20 ] , but hold it as a constant which is a function of the total number of data-points .

Summary: This paper study the problem of uncertainty quantification for the estimation error of the leading eigenvector from Oja's algorithm. The authors establish the distribution of $\sin^2$ error of the leading eigenvetor of Oja's algorithm. Also, the authors propose an online bootstrap algorithm for Oja's algorithm and provide the conditions under which the bootstrap is consistent.