Bootstrapping the Error of Oja's Algorithm 

1 Introduction . Since its discovery over a century ago [ 13 ] , principal component analysis ( PCA ) has been a cornerstone of data analysis . In many applications , dimension reduction is paramount and PCA offers an optimal low-rank approximation of the original data . PCA is also highly interpretable as it projects the dataset onto the directions that capture the most variance known as principal components . Important applications of PCA include image and document analysis , where the largest few principal components may be used to compress a large dimensional dataset to a manageable size without incurring much loss ; for a discussion of some other applications of PCA , see for example , [ 28 ] . In these settings , the original dimensionality , which could be the number of pixels in an image or the vocabulary size after removing stop-words , is in the tens of thousands . An offline computation of the principal components would require the computation of eigenvectors of the sample covariance matrix . However , in high-dimensional settings , storing the covariance matrix and subsequent eigen-analysis can be challenging . Streaming PCA methods have gained significant traction owing to their ability to iteratively update the principal components by considering one data-point at a time . One of the most widely used algorithms for streaming PCA is Oja ’ s algorithm , proposed in the seminal work of [ 41 ] . Oja ’ s algorithm involves the following update rule : wt+1 − wt = η ( wTt Xt ) Xt ; wTt+1wt+1 = 1 , ( 1 ) where Xt ∈ Rd is the tth data point and wt is the current estimate for the leading eigenvector of Σ = EXXT after t data-points have been seen . The parameter η can be thought of as a learning rate , which can either be fixed or varied as a function of t. In this paper we fix the learning rate , similar to [ 26 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Contribution : In the present work , we consider the problem of uncertainty quantification for the estimation error of the leading eigenvector from Oja ’ s algorithm , which is one of the most commonly used streaming PCA algorithms . Our contributions may be summarized as follows : 1 . We derive a high-dimensional weighted χ2 approximation to the sin2 error for the leading eigenvector of Oja ’ s algorithm . We recover the optimal convergence rate O ( 1/n ) while allowing d to grow at a sub-exponential rate under suitable structural assumptions on the covariance matrix , matching state-of-the-art theoretical results for consistency of Oja ’ s algorithm . Our result provides a distributional characterization of the sin2 error for Oja ’ s algorithm for the first time in the literature . The approximation holds for a wide range of step sizes . 2 . Since the weighted χ2 approximation depends on unknown parameters , we propose an online bootstrap algorithm and establish conditions under which the bootstrap is consistent . Our bootstrap procedure allows the approximation of important quantities such as the quantiles of the error associated with Oja ’ s algorithm for the first time . Prior analysis of Oja ’ s algorithm . While Oja ’ s algorithm was invented in 1982 it was not until recently that the theoretical workings of Oja ’ s algorithm have been understood . A number of papers in recent years have focused on proving guarantees of convergence of the iterative update in ( 1 ) toward the principal eigenvector of the ( unknown ) covariance matrix EXXT , which can be recast as stochastic gradient descent ( SGD ) on the quadratic objective function min w wTw=1 −trace ( wTΣw ) , Σ = EXXT , ( 2 ) projected onto the non-convex unit sphere . We assume that the data-points are mean zero . Despite being non-convex and thus falling outside the framework for which theory for stochastic gradient descent convergence is firmly established , the output of Oja ’ s algorithm be viewed as a product of random matrices and shares similar structure to other important classes of non-convex problems , such as matrix completion [ 27 , 29 ] , matrix sensing [ 27 ] , and subspace tracking [ 4 ] . Thus , studying this optimization problem serves as a natural first step toward understanding the behavior of SGD in more general non-convex settings . Let v1 denote the principal eigenvector of Σ , and let v̂1 = wn be the solution to the stochastic iterative method applying Eq 1 . Finally , let λ1 > λ2 be the first and second principal eigenvalues of Σ . Sharp rates of convergence for Oja ’ s updates were established in [ 25 ] . Under boundedness assumptions on ‖XiXTi − Σ‖ , they show that with constant probability , the square of the sine of the angle between v1 and w satisfies : 1− ( vT1 v̂1 ) 2 = O ( 1 n ) ( 3 ) where the O hides a constant which depends in the optimal way on the eigengap between the top two eigenvalues , and independent of n or d , improving on previous error bounds for Oja ’ s algorithm [ 46 , 18 , 3 , 47 , 38 , 2 ] which showed convergence rates that deteriorate with the ambient dimension d , and thus did not fully explain the efficiency of Oja ’ s update . This sharp rate is remarkable , as it matches the error of the principal eigenvector of the sample covariance matrix , which is the batch or offline version of PCA . Other notable work include [ 31 , 33 ] for unbounded Xi , analysis of Oja ’ s algorithm for computing top k principal components [ 1 , 24 ] . The bootstrap . The bootstrap , proposed by [ 9 ] , is one of the most widely used methods for uncertainty quantification in machine learning and statistics and accordingly has a vast literature . We refer the reader to [ 17 , 49 ] for expositions on the classical theory of the bootstrap for IID data . Recently , since the groundbreaking work of [ 7 , 8 ] , the bootstrap has seen a renewed surge of interest in the context of high-dimensional data where d can be potentially exponentially larger than n. Of particular relevance to the present work are high-dimensional central limit theorems ( CLTs ) for quadratic forms , which have been studied by [ 43 , 51 , 15 ] . In particular , our CLT for the estimation error of Oja ’ s algorithm invokes a modest adaptation of [ 51 ] to independent but non-identically distributed random variables . In machine learning , bootstrap methods have been used to estimate the uncertainty of randomized algorithms such as bagging and random forests [ 35 ] , sketching for large scale singular value decomposition ( SVD ) [ 36 ] , randomized matrix multiplication [ 37 ] , and randomized least squares [ 34 ] . A standard notion of bootstrap consistency is that , conditioned on the data , the distribution of the suitably centered and scaled bootstrap functional approaches the true distribution with high probability in some norm on probability measures , typically the Kolmogorov distance , which is the supremum of the absolute pointwise difference between two CDFs . Bootstrap consistency is often established by deriving a Gaussian approximation for the sampling distribution and showing that the bootstrap distribution is close to the corresponding Gaussian approximation with high probability . It may seem that if one knows that the approximating distribution of a statistic is Gaussian , this defeats the purpose of bootstrap . However , for most statistics , the parameters of the normal approximation depend on unknown model parameters , and have to be estimated if one intends to use the normal approximation . Furthermore , the CLT only gives a first-order correct approximation of the target distribution , i.e . with O ( 1/ √ n ) error . In contrast , the bootstrap of a suitably centered and scaled statistic has been shown to be higher order correct for many functionals [ 16 , 17 , 19 ] . Quantifying uncertainty for SGD . Behind the recent success of neural networks in a wide range of sub-fields of machine learning , the workhorse algorithm has become Stochastic gradient descent ( SGD ) [ 42 , 40 , 44 ] . For establishing consistency of bootstrap , one requires to establish asymptotic normality [ 11 , 42 , 45 , 39 ] . There has also been many works on uncertainty estimation of SGD [ 6 , 32 , 12 , 48 ] . However , all these works are for convex , and predominantly strongly convex loss functions . Only recently , [ 52 ] has established asymptotic normality for nonconvex loss functions under dissipativity conditions and appropriate growth conditions on the gradient , which are weaker conditions than strong convexity but not significantly so . Now , in Section 2 we present notation and do setup , present our main theoretical results in Section 3 , followed by simulations in Section 4 . 2 Preliminaries . We consider a row-wise IID triangular array , where the random vectors { Xi } in the nth row take values in Rdn , with E [ Xi ] = 0 and Var ( Xi ) = Σn . Note that the triangular array allows { X1 , . . . , Xn } come from a different distribution for each n and the setting where d is fixed and n grows is a special case . For readability , we drop the subscript n from Σn . We use ‖ · ‖ to denote the Euclidean norm for vectors and the operator norm for matrices and ‖ · ‖F to denote the Frobenius norm . Expanding out the recursive definition in Eq 1 , we see that Oja ’ s iteration can be expressed as wt+1 = ( Id + ηXtX T t ) wt . Thus , after n iterations the vector can be written as a matrix-vector product , where the matrix is a product of n independent matrices . Expanding out the recursive definition , we get : Bn : = n∏ i=1 ( Id + ηXiX T i ) v̂1 = Bnu0 ‖Bnu0‖ , ( 4 ) where Id is a d× d identity matrix . where u0 is a random unit vector in d dimensions . In the scalar case , when η = 1/n , for large n , the numerator of Eq 4 behaves like exp ( ∑ iX 2 i /n ) , which in turn converges to exp ( E [ X21 ] ) . For matrices , one hopes that , by independence , a result of the same flavor will hold . And in fact if it does hold , then for η = lognn , the numerator in Eq 4 will concentrate around exp ( log nΣ ) . The spectrum of this matrix is dominated by the principal eigenvector , i.e . the ratio of the first eigenvalue to the second one is exp ( log n ( λ1 − λ2 ) ) , where λi is the ith eigenvalue of the covariance matrix Σ . This makes it clear that Oja ’ s algorithm is essentially a matrix vector product of this matrix exponential ( suitably scaled ) and a random unit vector . However , the intuition from the scalar case is nontrivial to generalize to matrices due to noncommutativity . Limits of products of random matrices have been studied in mathematics in the context of ergodic theory on Markov chains ( see [ 14 , 30 , 5 , 10 ] etc. ) . However , until recent results of [ 23 ] , which extended and improved results in [ 21 ] , there has not been much work on quantifying the exact rate of convergence , or finite-sample large deviation bounds for how a random matrix product deviates from its expectation . We reparametrize η as ηn/n , where ηn is chosen carefully to obtain a suitable error rate . Note that this is not a scheme where we decrease η over time as in [ 20 ] , but hold it as a constant which is a function of the total number of data-points .

Summary: Oja’s principal subspace algorithm is a well-known and powerful technique for learning and tracking principal information in a time series setting. In this paper, the authors use U-statistics and a concentration of matrix products to obtain an error estimate between the leading eigenvector and the output of Oja’s algorithm. The other eigenvectors are probably beyond the theoretical understanding of random matrix theory at the moment. There are two main contributions: (1) Establishes the distribution of the sin^2 error estimate for the leading eigenvector computed via Oja’s algorithm and (2) A stream-friendly bootstrap algorithm to compute quantiles of the error estimate. The paper contains an interesting theoretical analysis involving modern matrix random theory, and since they have the distribution, a potential application to bootstrapping. The main novelties in this paper are theoretical, and the author could better demonstrate the advantages of their online multiplier bootstrapping technique.