Title: Streaming PCA for Markovian Data
Venue: NeurIPS 2023
Accepted: yes
Keywords: Streaming PCA

Markov Chain

Mixing

Oja's algorithm
Abstract: Since its inception in 1982, Oja's algorithm has become an established method for streaming principle component analysis (PCA). We study the problem of streaming PCA, where the data-points are sampled from an irreducible, aperiodic, and reversible Markov chain starting in stationarity. Our goal is to estimate the top eigenvector of the unknown covariance matrix of the stationary distribution. This setting has implications in scenarios where data can solely be sampled from a Markov Chain Monte Carlo (MCMC) type algorithm, and the objective is to perform inference on parameters of the stationary distribution. Most convergence guarantees for Oja's algorithm in the literature assume that the data-points are sampled IID. For data streams with Markovian dependence, one typically downsamples the data to get a "nearly" independent data stream. In this paper, we obtain the first near-optimal rate for Oja's algorithm on the entire data, where we remove the logarithmic dependence on the sample size, $n$, resulting from throwing data away in downsampling strategies.

Streaming PCA for Markovian Data

Abstract

Since its inception in 1982, Oja's algorithm has become an established method for streaming principle component analysis (PCA). We study the problem of streaming PCA, where the data-points are sampled from an irreducible, aperiodic, and reversible Markov chain starting in stationarity. Our goal is to estimate the top eigenvector of the unknown covariance matrix of the stationary distribution. This setting has implications in scenarios where data can solely be sampled from a Markov Chain Monte Carlo (MCMC) type algorithm, and the objective is to perform inference on parameters of the stationary distribution. Most convergence guarantees for Oja's algorithm in the literature assume that the data-points are sampled IID. For data streams with Markovian dependence, one typically downsamples the data to get a "nearly" independent data stream. In this paper, we obtain the first near-optimal rate for Oja's algorithm on the entire data, where we remove the logarithmic dependence on the sample size, n , resulting from throwing data away in downsampling strategies.

1 Introduction

Streaming Principal Component Analysis (PCA) is an important and well studied problem where the principal eigenvector of the sample covariance matrix of a dataset is computed one data-point at a time. One of the most popular algorithms for streaming PCA was introduced by Erkki Oja in 1982 [29, 30]. Most existing analyses of Oja's algorithm are done when the data is sampled IID.

However, in many practical applications, the data-points are dependent and are sampled from an MCMCprocess converging to a target stationary distribution. This naturally arises in the context of token algorithms for Federated PCA settings [10, 12, 13] with multiple machines communicating via a fixed and connected graph topology. Each machine contains an arbitrary fraction of data-points and the goal is to design a streaming algorithm that respects this topology and returns the principal component of the whole dataset. This is typically achieved using a Metropolis-Hastings scheme that uses local information to design the transition matrix of a Markov chain with any desired stationary distribution. The stationary distribution, π , of the random walk is chosen so that the distribution of the samples under π matches the uniform distribution over data-points. Governed by this Markov chain, a random walker then travels the network of machines and samples one data-point at a time from the current machine and computes the update. However, even under the stationary distribution, the data-points are dependent, which deviates from the IID setup. Our goal is to obtain a near-optimal analysis of the sin 2 error of the estimated vector with respect to the true top eigenvector of the unknown covariance matrix in the Markovian setting.

Estimating the first principal component with streaming PCA: Let X t be a mean zero d dimensional vector with covariance matrix Σ , and let η t be a decaying learning rate. The update rule of Oja's algorithm is given as -

where w t is the estimate of v 1 , the top eigenvector of Σ , and η t is the step-size at timestep t . We aim to analyse the sin 2 error of Oja's iterate at timestep t , defined as 1 -⟨ w t , v 1 ⟩ 2 .

Streaming PCA in the IID setting: For an IID data stream with E [ X i ] = 0 and E [ X i X T i ] = Σ , there has been a lot of work on determining the non-asymptotic convergence rates for Oja's algorithm and its various adaptations [16, 1, 3, 38, 14, 15, 26, 21, 25]. Amongst these, [16], [1] and [15] match the optimal offline sample complexity bound suggested by the independent and identically distributed (IID) version of Theorem 1 (See Theorem 1.1 in [16]).

We consider Oja's algorithm in the setting where the data is generated from a reversible, irreducible, and aperiodic Markov chain with stationary distribution π . We denote by E π [ . ] the expectation under the stationary distribution. In this setting our goal is to estimate the principal eigenvector of E π [ X i X T i ] . As in the IID setting, E π [ X i ] = 0 . The challenge is that the data, even when it reaches stationarity, is dependent. Here the degree of dependence is captured by the second eigenvalue in the magnitude of the transition matrix P (denoted as | λ 2 ( P ) | ) of the Markov chain. This is closely related to the mixing time of a Markov chain [19], denoted as τ mix, which is the time after which the conditional distribution of a state is close in total variational distance to its stationary distribution, π (See Section 2.1).

Our contribution: Using a series of approximations, we obtain an optimal error rate for the sin 2 error, which is worse by a factor of 1 / (1 -| λ 2 ( P ) | ) from the corresponding error rate of the IID case. Previous work [3] has established rates worse by a poly-logarithmic factor by using downsampling, i.e. applying the update on every k th datapoint. In Figure 1, we compare Oja's algorithm with its downsampled and offline variants (see Section 6 for more details on setup). We see that Oja's algorithm performs significantly better than the downsampled variant, and similarly to the offline variant where for the i th data point we compute the eigenvector of the sample covariance matrix of all data-points up-to i . Our work provides a concrete and novel result that explains these observations. In Table 1, we compare our bounds with related analyses of Oja's algorithm. The last row shows that we are the first to obtain an error whose main term is free of logarithmic dependence on n or d for streaming PCA in the Markovian case.

We break the logarithmic barrier in previous work by considering a series of approximations of finer granularity which uses reversibility of the Markov chain and standard mixing conditions of irreducible and aperiodic Markov chains. Our rates are comparable to the recent work of [28] (Proposition 1) that establishes an offline error analysis for estimating the principal component of the empirical covariance matrix of Markovian data by using a Matrix Bernstein inequality. Therefore, our results are nearly optimal in terms of the dependence on sample size, n , the dimension d , and the model parameters V , ( λ 1 -λ 2 ) and 1 -| λ 2 ( P ) | (See Section 2 for definitions). Our results also imply a linearly convergent decentralized algorithm for streaming PCA in a distributed setting. As a simple byproduct of our theoretical result, we also obtain a rate for Oja's algorithm applied on downsampled data, which is worse by a factor of log n , as shown in Figure 1. To our knowledge, this is the first work that analyzes the Markovian streaming PCA problem without any downsampling that matches the error of the offline algorithm.

The crux of our analysis uses the mixing properties of the Markov chain. Strong mixing intuitively says that the conditional distribution of a state s in timestep k given the starting state is exponentially close to the stationary distribution of s , the closeness being measured using the total variation distance. All previous work on Markovian data exploits this property by conditioning on states many time steps before. However, it is crucial to a) adaptively find how far to look back and b) bound the error of the sequence of matrices we ignore between the current state and the state we are conditioning on. Observe that these two components are related. Looking back too far makes the dependence very small but increases the error resulting from approximating a larger matrix product of intermediate matrices. We present a fine analysis that balances these two parts and then uses spectral theory to bound the second part within a factor of a variance parameter that characterizes the variability of the matrices and shows up in the analysis of [16, 28].

Related work on streaming PCA and online matrix decomposition on Markovian data: Amongst recent work, [3] is very relevant to our setting, since it analyzes Oja's algorithm with Markovian Data samples. Inspired by the ideas of [8], the authors propose a downsampled version of Oja's algorithm to reduce dependence amongst samples and provide a Stochastic Differential Equation (SDE) based analysis to achieve a sample complexity of O ( G ( λ 1 -λ 2 ) 2 1 ϵ log 2 ( G ( λ 1 -λ 2 ) 2 1 ϵ )) for sin 2 error smaller than ϵ , where G is a variance parameter. We obtain a similar rate in Corollary 1 through our techniques. However, comparing with Theorem 1, we observe that downsampling leads to an extra O (log ( n )) factor. It is important to point out that [3] provides an analysis for estimating top k principal components, whereas this paper focuses on obtaining a log -free error rate for the first principal component.

[22] consider the harder problem of online non-negative matrix factorization for Markovian data. Their analysis establishes asymptotic convergence of error, but does not provide a rate.

Stochastic Optimization with Markovian Data : Markovian models are often considered in Reinforcement Learning and Linear Dynamic Systems[2, 5, 9, 31, 4, 35, 18, 24]. There have been many notable nonasymptotic bounds for stochastic gradient descent (SGD) methods for general convex and nonconvex functions with Markovian data [8, 32, 6, 7, 10, 39, 34]. The convergence rates (sample complexities) obtained in these works apply to more general problems but do not exploit the matrix product structure inherent to Oja's algorithm. In this work, we develop novel techniques to show that a sharper analysis is possible for the PCA objective. The paper is organized as follows. Section 2 contains the problem setup and preliminaries about Markov Chains. Section 3 contains Theorem 1. We present a sketch of the

1.0

.......

Downsampled

Offline

Oja's algorithm

0.8

0.6

0.4

0.2

0.0

0

1000

2000

3000

4000

5000

main technical tools in Section 4, intermediate theorems needed for the main theorem in Section 5 and conclude with simulations in Section 6.

2 Problem Setup and Preliminaries

This section presents the problem setup and outlines important properties of the Markov chain that will be utilized subsequently. We assume that:

Assumption 1. The Markov chain is irreducible, aperiodic, reversible, and starts in stationarity, with state distribution π 1 .

Such a Markov chain can arise in various situations, for e.g., while performing random walks on expander graphs which are used extensively in fields such as computer networks, error-correcting codes, and pseudorandom generators. Each state s of the Markov chain is associated with a distribution D ( s ) over d -dimensional vectors with mean µ s ∈ R d and covariance matrix Σ s ∈ R d × d .

For a random walk s 1 , s 2 , · · · s t on the Markov chain, we define the sequence of random variables X 1 , X 2 · · · X t , where conditioned on the state s i , X i ∼ D ( s i ) .We represent the mean as µ := E s ∼ π [ µ s ] = ∑ s π s µ s and the covariance matrix as Σ ∈ R d × d , which, for i ∈ [ n ] can be expressed as:

In this work, we assume µ = 0 , which is a common assumption in the IID setting (see [16, 1]). While it may be possible to extend our analysis to the non-zero mean case, it is out of the scope of this paper. Therefore, Σ = E s i ∼ π E D ( s i ) [ X i X T i ] for i ∈ [ n ]

Let the eigenvalues of Σ be denoted as λ 1 > λ 2 ≥ λ 3 · · · λ d . Let v 1 denote the leading eigenvector of Σ and V ⊥ denote the R d × ( d -1) matrix with the remaining eigenvectors as columns. In this work, ∥ . ∥ 2 denotes the Euclidean L 2 norm for vectors and the operator norm for matrices unless otherwise specified and I denotes the identity matrix of the appropriate dimensionality. We proceed under the following standard assumptions for i ∈ [ n ] , (see for eg. [15]).

Assumption 3. ∥ X i X T i -Σ ∥ 2 ≤ M with probability 1.

Assumption 3 also implies ∥ Σ s + µ s µ T s -Σ ∥ 2 ≤ M with probability 1. WLOG, we assume M + λ 1 ≥ 1 . We use E [ . ] := E s ∼ π E D ( s ) [ . ] to denote the expectation over state s ∼ π and over the state-specific distributions D ( . ) , unless otherwise specified. We observe that Assumption 3 can be relaxed to accommodate sub-gaussian data distributions with appropriate variance decay (see Proposition 1 in [21]) using standard truncation-based arguments. We do not pursue this here. Define the matrix product

Unrolling the recursion in Eq 1, the output of Oja's algorithm at timestep t is given as w t = B t w 0 ∥ B t w 0 ∥ 2 .

2.1 Markov chain mixing times

Now we will discuss some well-known properties of an irreducible, aperiodic, and reversible Markov chain (see [19] for details and derivations). Let | λ 2 ( P ) | denote the second largest absolute eigenvalue of the Markov chain; let the state-distribution of the Markov chain at timestep t with s 1 = x be P t ( x, . ) . For any two probability distributions ν 1 and ν 2 , recall that the total variational distance is TV ( ν 1 , ν 2 ) := ∥ ν 1 -ν 2 ∥ TV := 1 2 ∑ x ∈ Ω | ν 1 ( x ) -ν 2 ( x ) | . The distance from π at the t th timestep is defined as d mix ( t ) := sup x ∈ Ω TV ( P t ( x, . ) , π ) . For irreducible and aperiodic Markov chains, by Theorem 4.9 in [19], we have d mix ( t ) ≤ C exp( -ct ) for some C, c > 0 . The mixing time of the Markov chain is defined as:

As defined in Section 4.5 of [19], we will denote τ mix := τ mix ( 1 4 ) . Then, we have τ mix ( ϵ ) ≤ ⌈ log 2 (1 /ϵ ) ⌉ τ mix. It is worth mentioning the useful relationship between d mix ( . ) and τ mix, given as d mix ( lτ mix ) ≤ 2 -l ∀ l ∈ N 0 . These results about mixing time are valid for general irreducible and aperiodic Markov chains. A reversible Markov chain satisfies ∀ x, y ∈ Ω , π ( x ) P ( x, y ) = π ( y ) P ( y, x ) . For a reversible, irreducible, and aperiodic Markov chain, the gap 1 -| λ 2 ( P ) | , is inversely proportional to (Theorem 12.4, τ mix [19]).

3 Main Results

In this section, we present our main result, a near-optimal convergence rate for Oja's algorithm on Markovian data. As a corollary, we also establish a rate of convergence for Oja's algorithm applied

on downsampled data, where every k th data-point is considered. Supplement Section S.5 contains comprehensive proofs of Theorem 1 and Corollary 1 while the proof of Proposition 1 can be found in Supplement Section S.2.

Theorem 1. Fix a δ ∈ (0 , 1) and let the step-sizes be η i := α ( λ 1 -λ 2 )( β + i ) with η 0 ≤ 1 e , α > 2 . Under assumptions 1, 2 and 3, for sufficiently large number of samples n such that n log ( 1 ηn ) > β log ( 1 η 0 ) ,

the output w n of Oja's algorithm (1) satisfies

with probability atleast (1 -δ ) . Here C is an absolute constant and

Remark 1. (Interpreting the sin 2 error) Theorem 1 simply establishes that, under suitable choices of α, β , as long as assumptions 1, 2 and 3 hold, for decaying step sizes η i = α ( λ 1 -λ 2 )( β + i ) , with probability at least 3 4 , we have,

for large enough n . This can be seen by using α = 6 , setting β as in Eq 4, and taking n = ˜ Ω ( d 0 . 1 ( β ) 1 . 2 ) , where ˜ Ω hides logarithmic factors in n . These choices make the first and third terms of the upper bound on the sin 2 error in Theorem 1 a smaller order than the second term, which is O ( 1 n ) . For the first term, we have d ( 2 β n ) 2 α = O ( 1 n 2 ) . The third term involves τ mix ( η 2 n ) n 2 , which is O ( log 2 ( n ) n 2 ) , using the definition of τ mix ( ϵ ) in Eq 3.

Remark 2. (Knowledge of mixing times) We note that setting the step-sizes η i in Theorem 1 requires the knowledge of the mixing time of the Markov chain, τ mix. An offline algorithm for estimating the mixing time is provided in [37]. They show that at least a constant τ mix | Ω | ϵ 2 samples are needed for estimating τ mix within an absolute error ϵ . Therefore, it is possible to use the first O ( 1 ϵ 2 ) samples to estimate the mixing time and then set the learning rate accordingly.

Remark 3. (Regarding non-reversible Markov chains) We assume the reversibility of Markov chains to simplify our analyses. Our techniques and results extend to the non-reversible case using reversibilization tools (see [20], [11]). Specifically, one can obtain bounds in terms of the second-largest eigenvalue, in magnitude, of the modified transition matrix P ∗ P , the multiplicative reversibilization of P . Here, P ∗ represents the probability transition matrix of the time-reversed chain, given as ∀ x, y ∈ Ω , P ∗ ( x, y ) = π ( y ) π ( x ) P ( y, x ) . We leave this extension for future work.

Next, we compare the rate of convergence proposed in Theorem 1 with the offline algorithm having access to the entire dataset { X i } n i =1 using a recent result from [28]. Here, the authors extend the Matrix Bernstein inequality [36, 33], to Markovian random matrices. Their setup is much like ours except that the matrix at any state is fixed, i.e., there is no data distribution D ( s ) as in our setup. However, it is easy to extend their result to our setting by observing that conditioned on the state sequence, the matrices X i X T i , i ∈ [ n ] are independent under our model, and we can push in the expectation over the state-specific distributions, D ( s ) , whenever required. Therefore, we have the following result -

Proposition 1 (Theorem 2.2 of [28]+Wedin's theorem) . Fix δ ∈ (0 , 1) . Consider an irreducible and aperiodic Markov chain. Under assumptions 2 and 3, with probability 1 -δ , the leading eigenvector ˆ v of ∑ n i =1 X i X T i /n satisfies,

for absolute constants C ′ 1 and C ′ 2 .

Observe that Theorem 1 matches the leading term V ( λ 1 -λ 2 ) 2 (1 -| λ 2 ( P ) | ) in Theorem 1 except the log( d ) term. We believe, much like the IID case (also see footnote 1 in [16]), this logarithmic term in [28]'s result is removable for large n and a constant probability of success.

Remark 4. (Comparison with IID algorithm) Fix a δ ∈ (0 , 1) . If the data-points { X i } n i =1 are sampled IID from the stationary distribution π , then under assumptions 2 and 3, using Theorem 4.1 from [16], we have that the output w n of Oja's algorithm 1 satisfies,

The leading term of Theorem 1 is worse by a factor of 1 1 -| λ 2 ( P ) | . Further, it has an additive lower order term O ( log 2 ( n ) n 2 ) due to the covariance between data-points in the Markovian case.

Corollary 1. (Downsampled Oja's algorithm) Fix a δ ∈ (0 , 1) . If Oja's algorithm is applied on the downsampled data-stream with every k th data-point, where k := τ mix ( η 2 n ) then under the conditions of Theorem 1 with appropriately modified α and β , the output w n satisfies,

with probability atleast (1 -δ ) . Here C is an absolute constant and C 1 := 30 α 2 2 α -1 , C 2 := 35 α 3 α -1 .

Remark 5. Data downsampling to reduce dependence amongst samples has been suggested in recent work [27, 23, 3]. In Corollary 1, we establish that the bound on the rate obtained is sub-optimal compared to Theorem 1 by a log ( n ) factor. We prove this by a simple yet elegant observation: the downsampled data stream can be considered to be drawn from a Markov chain with transition kernel P k ( ., . ) since each data-point is k steps away from the previous one. For sufficiently large k , this implies that the mixing time of this chain is Θ(1) . These new parameters are used to select the modified values of α, β according to Lemma S.11 in the Supplement.

The proof of Theorem 1 follows the same general recipe as in [16] for obtaining a bound on the sin 2 error. However, the original proof techniques heavily rely on the IID setting. We carry out a refined analysis for each step under the Markovian data model by a careful control of error terms arising out of dependence. The first step involves obtaining a high-probability bound on the sin 2 error, by noting that Oja's algorithm on n data-points can be viewed as a single iteration of the power method on B n . Therefore, fixing a δ ∈ (0 , 1) using Lemma 3.1 from [16], we have with probability at least (1 -δ ) ,

where C is an absolute constant. The numerator is bounded by first bounding its expectation (see Theorem 3) and then using Markov's inequality. To bound the denominator, similar to [16], we will use Chebyshev's inequality. Theorem 4 provides a lower bound for the expectation E [ v T 1 B n B T n v 1 ] . Chebyshev's inequality also requires upper-bounding the variance of E [ v T 1 B n B T n v 1 ] , which requires us to bound E [ ( v T 1 B n B T n v 1 ) 2 ] (see Theorem 5).

4 Main Technical Tools

In this section, we provide a sketch of the main arguments used in our proof.

Warm-up with downsampled Oja's algorithm: We start with the simple downsampled Oja's algorithm to build intuition. Here, one applies Oja's update rule (Eq 1) to every k th data-point, for a suitably chosen k . For k = ⌈ Lτ mix log n ⌉ , the total variation distance between any consecutive data-points in the downsampled data stream is O ( n -L ) . As we show in Corollary 1, the error of this algorithm is similar to the error of Oja's algorithm applied to n/k data-points in the IID setting, i.e., O ( V τ mix log n/n ) .

We will take E [ v T 1 B n B T n v 1 ] as an example. Let us introduce some notation.

We peel this quantity one matrix at a time from the inside. Note that for a reversible Markov chain, standard results imply (see Lemma 1) that the mixing conditions apply to the conditional distribution of a state given another state k steps in the 'future' (see Supplement Section S.3 for a proof). Recall d mix ( k ) from Section 2.1.

It will be helpful to explain our analysis by comparing it with the IID setting. For this reason, we will use E IID [ . ] to denote the expectation under the IID data model. Define α n,i := E [ v T 1 B n,i B T n,i v 1 ] , i ∈ [ n ] . Then we have,

where the first term is smaller than (1 + η 1 λ 1 ) 2 α n, 2 . We define T 1 and T 2 as follows. T 1 := E [ v T 1 B n, 2 ( I + η 1 Σ) ( X 1 X T 1 -Σ ) B T n, 2 v 1 ] , and T 2 := E [ v T 1 B n, 2 ( X 1 X T 1 -Σ ) 2 B T n, 2 v 1 ] .

For the IID setting, the second term is zero , and the third term can be bounded as follows:

Let us denote the IID version of α n,i by α IID n,i := E IID [ v T 1 B n,i B T n,i v 1 ] . The final recursion for the IID case becomes: α IID n, 1 ≤ (1 + 2 η 1 λ 1 + η 2 1 ( λ 2 1 + V ) ) α IID n, 1 . So, for our Markovian data model, the hope is that the cross term T 1 (which has a multiplicative factor of η 1 ) is O ( η 1 ) and T 2 is O ( η 2 1 ) . We will start with the T 1 term, which is zero in the IID setting.

Figure 2: If we could replace the intermediate products (white matrices) by I , the conditional expectation of the noise matrix X 1 X T 1 -Σ conditioned on the gray matrices would be nearly zero.

We will show that T 1 , while not zero like the IID case, is still sufficiently small. Intuitively, if we could replace the k matrices between X 1 X T 1 -Σ and B n, 2+ k for some suitably large integer k by identity (see Figure 2), then using (reverse) mixing properties of the Markov chain, we could argue using Lemma 1 that E [ X 1 X T 1 -Σ | s 1+ k , . . . , s n ] is very close to zero (see Figure 2). The following lemma formally bounds the deviation of the lengthk matrix product from identity.

Lemma 2. Let Assumption 3 hold. If ∀ i ∈ [ n ] , η i k i ( M + λ 1 ) ≤ ϵ, ϵ ∈ (0 , 1) and η i forms a non-increasing sequence then ∀ m ≤ n -k n ,

Lemma 2 bounds the norm of the matrix product B t + k t -1 ,t at two levels. The first result provides a coarse bound, approximating linear and higher-order terms. The second result provides a finer bound, preserving the linear term and approximating quadratic and higher-order terms. The proofs involve a straightforward combinatorial expansion of B t + k t -1 ,t and are deferred to the Supplement Section S.3.

Approximating ∏ k +1 i =2 ( I + η i X i X T i ) requires η 1 k to be small. Since this is a recursive argument, we would need η i k to be small for i = 1 , . . . n , which is satisfied by the strong condition η 1 k is small. To obtain a tight analysis, we choose k adaptively. We set k i = τ mix ( η 2 i ) (see definition in Eq 3).

As we will show in detail in the Supplement, Lemma 2 Eq 10 along with the adaptive choice of k i gives us a sharp error bound. Using it, we can bound T 1 (see Eq 8) as:

Naively bounding the T 1 ,j term by O (1) leads to the same rate as downsampled Oja's algorithm.

In the following lemma, we will establish that, indeed, T 1 ,j has a much smaller norm. The novelty of our bound is not just in using the mixing properties of the Markov chain but also in teasing out the variance parameter V . We will state the lemma, in a slightly more general form as -

Lemma 3. Under Assumptions 1, 2 and 3, for i < j ≤ i + k i ,

where k i is as defined in Lemma S.11 and S is a constant symmetric positive semi-definite matrix.

Lemma 3 bounds the norm of the covariance between matrices ( X i X T i -Σ ) S and X j X T j . In particular, this implies that the norm of T 1 ,j decays as | λ 2 ( P ) | j -1 . The proof uses a spectral argument that replaces a coarse approximation by a sum of k i O (1) terms to sum of k exponentially decaying terms, thereby removing the dependence on k i , which can be as large as log( n ) . The proof is deferred to the Supplement Section S.4.

Let { c 1 , c 2 , c 3 , c 4 } be positive constants for ease of notation. Coming back to Eq 8, we can bound T 1 as follows: T 1 ≤ α n,k +2 ( η 1 c 1 | λ 2 ( P ) |V 1 -| λ 2 ( P ) | + c 2 η 2 1 k 2 1 ) . A similar argument can be applied to bound

T 2 as: T 2 ≤ α n,k +2 ( V + c 3 η 1 k 2 1 ) . Putting everything together in 8, we have

Recursing on this inequality gives us our bound on E [ v T 1 B n B T n v 1 ] (Theorem 2). We are now ready to present all our accompanying theorems.

5 Intermediate Theorems for Convergence Analysis

In this section, we present our accompanying theorems which are used to obtain the main result in Theorem 1. But before doing so, we will need to establish some notation. Let k i := τ mix ( η 2 i ) , and the step-sizes be set as η i := α ( λ 1 -λ 2 )( β + i ) with α, β as defined in Theorem 1. Let ϵ := 1 100 . As shown in Lemma S.11 in Supplement Section S.3 our choice of step-sizes satisfy, ∀ i ∈ [ n ] ,

Further, we define scalar variables -

and recall the definitions of B t and B j,i in Eqs 2 and 7, respectively. We are now ready to present the theoretical results needed to prove our main result. For simplicity of notation, we present versions of the results by using η i := α ( λ 1 -λ 2 )( β + i ) with α, β as defined in Theorem 1. However, these theorems are in fact valid under more general step-size schedules. We state and prove the more general versions in the Supplement Section S.4.

Theorem 2. Under Assumptions 1, 2 and 3, for all n > k n , and η i satisfying C.1 and C.2 , we have,

The three primary differences with the IID case are a) the (1 + r ) 2 term, which arises since the recursion sketched in Section 4 leaves out the last k n terms which are bounded by (1 + r ) 2 ; (b) the new factor of 1 1 -| λ 2 ( P ) | with V due to the Markovian dependence between terms; and c) the extra lower order term η 3 t ψ k,t arising from the use of Lemmas 2 and 3.

Theorem 3. Let u := min { t : t ∈ [ n ] , t -k t ≥ 0 } . Under Assumptions 1, 2 and 3, for all n > u , and η i satisfying C.1 and C.2 ,

Here, the difference is mainly in the new variable u , arising since the recursion stops at u , not 1 . (1 + 5 ϵ ) represents the approximation of the first u terms.

Theorem 4. Under Assumptions 1, 2 and 3, for all n > k n , η i satisfying C.1 and C.2 , and s := 2 r + δ 1000 , we have,

This differs from its IID counterpart by a multiplicative factor of (1 -s ) for the same reason as before, which also makes the sums go up to ( n -k n ) instead of n . Note that for sufficiently large n (Lemma S.12), r = O ( log( n ) n ) is very small and δ ∈ (0 , 1) . Therefore, (1 -s ) ≈ 1 as large n .

Theorem 5. Under Assumptions 1, 2 and 3, for all n > k n , and η i satisfying C.1 and C.2 , we have,

The differences are similar to the last theorems involving v 1 . Surprisingly, for this, the coarse approximation suffices, leading to an absence of the V term in the bound. Having established these results, the final step is to substitute them into Eq 6 and follow the proof recipe described earlier. This requires significant calculations and is deferred to the Supplement Section S.5.

6 Experimental Validation

In this section, we present some simple experiments to validate our theoretical results. For more detailed experiments, see the Supplement Section S.6. We design a Markov chain with | Ω | = 10 states, where the transition matrix entries P ij equal ρ/ ( | Ω |-1) for i = j and 1 -ρ for i = j . Smaller values of ρ lead to larger mixing times. It can be verified that the stationary distribution π = U (Ω) is uniform over the state-space and | λ 2 ( P ) |≈ (1 -ρ ) . We set ρ = 0 . 2 for Figures 1 and 3a, and vary it in Figure 3b. Each point in the plot is averaged over 20 random runs over different Markov chains, datasets, and initialization.

̸

(a) Variation of

2

error with

β

1.0

β = 1.0

1.0

|入(P)| =1 -5e-1

β = 0.8

|入(P)| = 1 - 5e-3

β = 0.6

[入(P)| = 1 - 1e-3

0.8

β =0.4

0.8

|入(P)| = 1 - 5e-4

0.6

0.6

0.4

0.4

0.2

0.2

0.0

0.0

0

1000

2000

3000

4000

5000

0

1000

2000

3000

4000

5000

Each state s ∈ Ω is associated with D ( s ) := Bernoulli( p s ) distribution. We set d = 1000 and select p s ∼ U (0 , 0 . 05) at the start of each random run. The covariance matrix, Σ s , for each state is set as Σ s ( i, j ) = exp ( -| i -j | c s ) σ i σ j where c s := 1 + 9 ( s -1 | Ω |-1 ) , σ i := 5 i -β . We start with the stationary distribution π , and for each state s i , we draw IID samples Z i ∼ D ( s i ) . We standardize Z i such that all components have zero mean and unit variance under the state distribution, D ( s i ) . We then generate the sample data-point for PCA as X i = Σ 1 2 i Z i . By construction, E D ( s i ) [ X i X T i ] = Σ i and E [ X i ] = 0 d . The step sizes for Oja's algorithm are set as η i = α ( β + i )( λ 1 -λ 2 ) for α = 5 , β = 5 1 -| λ 2 ( P ) | . For the downsampled variant, every 10 th data-point is considered, and β is accordingly divided by 10. For the offline algorithm, we recompute the leading eigenvector of the sample covariance matrix of data-points seen so far.

Figure 1 compares the performance of different algorithms for the Bernoulli distribution. Here, we are checking if the results obtained in Theorem 1, Proposition 1, and Corollary 1 are reflected in the experiments. The experimental results demonstrate that Oja's algorithm performs significantly better than the downsampled version, consistent with the theoretical results. It also shows that Oja's algorithm performs similarly to the offline algorithm, which is also confirmed by our theoretical results and that of [28]. Figure 3a compares the performance of Oja's algorithm for different covariance matrices. Smaller values of β decrease the eigengap λ 1 -λ 2 , and hence lead to a slower convergence. Figure 3b confirms that smaller values of ρ (larger values of | λ 2 ( P ) | ) also worsen the rate, which matches with our theoretical results.

7 Conclusion

We have considered the problem of streaming PCA for Markovian data, which has implications in various settings like decentralized optimization, reinforcement learning, etc. The analysis of streaming algorithms in such settings has seen a renewed surge of interest in recent years. However, the dependence between data-points makes it difficult to obtain sharp bounds. We provide, to our knowledge, the first near-optimal bound for obtaining the first principal component from a Markovian data stream that breaks the logarithmic barrier present in the analysis done for downsampled data. We believe that the theoretical tools that we have developed in this paper would enable one to obtain sharp bounds for other dependent data settings, learning top k principal components, and online inference algorithms with updates involving products of matrices.

References

Zeyuan Allen-Zhu and Yuanzhi Li. First efficient convergence for streaming k-pca: a global, gap-free, and near-optimal rate. In 2017 IEEE 58th Annual Symposium on Foundations of Computer Science (FOCS) , pages 487-492. IEEE, 2017.

Jalaj Bhandari, Daniel Russo, and Raghav Singal. A finite time analysis of temporal difference learning with linear function approximation. CoRR , abs/1806.02450, 2018.

Minshuo Chen, Lin Yang, Mengdi Wang, and Tuo Zhao. Dimensionality reduction for stationary time series via stochastic nonconvex optimization. Advances in Neural Information Processing Systems , 31, 2018.

Shuhang Chen, Adithya Devraj, Ana Busic, and Sean Meyn. Explicit mean-square error bounds for monte-carlo and linear stochastic approximation. In International Conference on Artificial Intelligence and Statistics , pages 4173-4183. PMLR, 2020.

Thinh T. Doan, Lam M. Nguyen, Nhan H. Pham, and Justin Romberg. Convergence rates of accelerated markov gradient descent with applications in reinforcement learning, 2020.

Thinh T. Doan, Lam M. Nguyen, Nhan H. Pham, and Justin Romberg. Finite-time analysis of stochastic gradient descent under markov randomness, 2020.

Ron Dorfman and Kfir Yehuda Levy. Adapting to mixing time in stochastic optimization with markovian data. In International Conference on Machine Learning , pages 5429-5446. PMLR, 2022.

John C. Duchi, Alekh Agarwal, Mikael Johansson, and Michael I. Jordan. Ergodic mirror descent. SIAM J. Optim. , 22(4):1549-1578, 2012.

Alain Durmus, Eric Moulines, Alexey Naumov, Sergey Samsonov, and Hoi-To Wai. On the stability of random matrix product with markovian noise: Application to linear stochastic approximation and td learning. In Conference on Learning Theory , pages 1711-1752. PMLR, 2021.

Mathieu Even. Stochastic gradient descent under markovian sampling schemes, 2023.

James Allen Fill. Eigenvalue Bounds on Convergence to Stationarity for Nonreversible Markov Chains, with an Application to the Exclusion Process. The Annals of Applied Probability , 1(1):62 - 87, 1991.

Andreas Grammenos, Rodrigo Mendoza-Smith, Cecilia Mascolo, and Jon Crowcroft. Federated PCA with adaptive rank estimation. CoRR , abs/1907.08059, 2019.

Anne Hartebrodt, Reza Nasirigerdeh, David B. Blumenthal, and Richard Röttger. Federated principal component analysis for genome-wide association studies. In 2021 IEEE International Conference on Data Mining (ICDM) , pages 1090-1095, 2021.

Amelia Henriksen and Rachel Ward. AdaOja: Adaptive Learning Rates for Streaming PCA. arXiv e-prints , page arXiv:1905.12115, May 2019.

De Huang, Jonathan Niles-Weed, and Rachel Ward. Streaming k-pca: Efficient guarantees for oja's algorithm, beyond rank-one updates. CoRR , abs/2102.03646, 2021.

Prateek Jain, Chi Jin, Sham Kakade, Praneeth Netrapalli, and Aaron Sidford. Streaming pca: Matching matrix bernstein and near-optimal finite sample guarantees for oja's algorithm. In Proceedings of The 29th Conference on Learning Theory (COLT) , June 2016.

Chi Jin, Sham M Kakade, Cameron Musco, Praneeth Netrapalli, and Aaron Sidford. Robust shift-and-invert preconditioning: Faster and more sample efficient algorithms for eigenvector computation. arXiv preprint arXiv:1510.08896 , 2015.

Harold J Kushner and G George Yin. Applications in signal processing, communications, and adaptive control. Stochastic Approximation and Recursive Algorithms and Applications , pages 63-93, 2003.

David A Levin and Yuval Peres. Markov chains and mixing times , volume 107. American Mathematical Soc., 2017.

Pascal Lezaud. Chernoff-type bound for finite Markov chains. The Annals of Applied Probability , 8(3):849 - 867, 1998.

Robert Lunde, Purnamrita Sarkar, and Rachel Ward. Bootstrapping the error of oja's algorithm. Advances in Neural Information Processing Systems , 34:6240-6252, 2021.

Hanbaek Lyu, Deanna Needell, and Laura Balzano. Online matrix factorization for Markovian data and applications to Network Dictionary Learning. arXiv e-prints , page arXiv:1911.01931, November 2019.

Shaocong Ma, Ziyi Chen, Yi Zhou, Kaiyi Ji, and Yingbin Liang. Data sampling affects the complexity of online sgd over dependent data. In Uncertainty in Artificial Intelligence , pages 1296-1305. PMLR, 2022.

Abdelkader Mokkadem. Mixing properties of arma processes. Stochastic Processes and their Applications , 29(2):309-315, 1988.

Jean-Marie Monnez. Stochastic approximation of eigenvectors and eigenvalues of the qsymmetric expectation of a random matrix. Communications in Statistics-Theory and Methods , pages 1-15, 2022.

Nikos Mouzakis and Eric Price. Spectral guarantees for adversarial streaming pca, 2022.

Dheeraj Nagaraj, Xian Wu, Guy Bresler, Prateek Jain, and Praneeth Netrapalli. Least squares regression with markovian data: Fundamental limits and algorithms. Advances in neural information processing systems , 33:16666-16676, 2020.

Joe Neeman, Bobby Shi, and Rachel Ward. Concentration inequalities for sums of markov dependent random matrices, 2023.

Erkki Oja. Simplified neuron model as a principal component analyzer. Journal of Mathematical Biology , 15(3):267-273, November 1982.

Erkki Oja and Juha Karhunen. On stochastic approximation of the eigenvectors and eigenvalues of the expectation of a random matrix. Journal of mathematical analysis and applications , 106(1):69-84, 1985.

Rayadurgam Srikant and Lei Ying. Finite-time error bounds for linear stochastic approximation andtd learning. In Conference on Learning Theory , pages 2803-2830. PMLR, 2019.

Tao Sun, Yuejiao Sun, and Wotao Yin. On markov chain gradient descent. Advances in neural information processing systems , 31, 2018.

Joel A Tropp. User-friendly tail bounds for sums of random matrices. Foundations of computational mathematics , 12:389-434, 2012.

Lan V Truong. Generalization error bounds on deep learning with markov datasets. Advances in Neural Information Processing Systems , 35:23452-23462, 2022.

John Tsitsiklis and Benjamin Van Roy. Analysis of temporal-diffference learning with function approximation. In M.C. Mozer, M. Jordan, and T. Petsche, editors, Advances in Neural Information Processing Systems , volume 9. MIT Press, 1996.

Roman Vershynin. Introduction to the non-asymptotic analysis of random matrices. arXiv preprint arXiv:1011.3027 , 2010.

Geoffrey Wolfer and Aryeh Kontorovich. Estimating the mixing time of ergodic markov chains. In Alina Beygelzimer and Daniel Hsu, editors, Proceedings of the Thirty-Second Conference on Learning Theory , volume 99 of Proceedings of Machine Learning Research , pages 3120-3159. PMLR, 25-28 Jun 2019.

Puyudi Yang, Cho-Jui Hsieh, and Jane-Ling Wang. History pca: A new algorithm for streaming pca. arXiv preprint arXiv:1802.05447 , 2018.

Ingvar Ziemann and Stephen Tu. Learning with little mixing. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 4626-4637. Curran Associates, Inc., 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe submission considers the task of *streaming* PCA, estimating the largest eigenvector of a covariance matrix in an online manner as new data points arrive. The focus is on theoretical analysis of a classical method, Oja's Algorithm, in the hitherto unconsidered non-IID setting. \n\nA seemingly strong restriction is made that the data come from a chain that is not only ergodic, but also reversible. The latter seems a potentially strong condition, but the justification appears as this is exactly the setting arising in algorithms in Federated PCA. \n\nThe primary contribution appears to be a theoretical justification for using Oja's algorithm as per normal in the considered setting, as opposed to \"downsampling\" points to decrease dependence between them. A multiplicative factor of how optimal error rates differ in the setting considered from the IID setting is also established.\n\nSTRENGTHS:\nThe theoretical problem appears to be tackled in a manner involving some level of sophistication, and the results of potential interest. As the authors themselves state, many intermediate results (as contained in the supplement) may be of wider interest. \n\nThat said, whilst I am familiar with many of the individual constituent elements of this work, I am not familiar with the dependent streaming PCA literature so much, so it is difficult for me to assess the contribution.\n\nWEAKNESSES:\n* The submission is, on the whole, not particularly well written in my opinion. It could benefit from another round of two of editing and polish. The implications of *why* the particular problem are being studied and clearer discussions of its potential use would be beneficial. There are also some obvious typos and undefined notations.\n\n* The problem considered appears quite niche and it is not made clear in the paper the extent to which the results are of wider interest (a cursory mention is made of reinforcement learning in the conclusion, but there is no discussion of this).\n\n* As the motivation for the paper is for settings where the output comes from MCMC algorithms, I would expect an experiment exploring the effect of different burn in periods and how well Oja's method performs in practical settings and when the chain has converged to what is generally considered close enough to its stationary distribution for iterates to be used in practice (see comment below about burn in). \n\n* As far as I can tell, simply downsampling as done in the experiment is not equivalent to the cited Chen et al. (2018) approach, who propose further refinements to the algorithm (see their Algorithm 1). As the papers primary contribution is supposedly justification for using Oja's algorithm as is over other approaches for dependent data, I would expect at a minumum a comparison is made to the actual Chen algorithm (this includes for the experiments in the supplement), as well as any other recent competitors. \n\n* The assumption of requiring the data come from a reversible process appears somewhat strong to be of general interest. I understand the specific application, but the authors should discuss how/if this holds in a wider setting. That is, under what conditions are multivariate time series reversible? Does this hold for certain wide cases of interest? \n\n* Issue: Page 1, footnote: It is not a fair assumption to assume the assumption holds after a sufficient period of burn in. Indeed, a burn in period may suffice to ensure that a Markov chain has burned in long enough to ensure future iterates have distribution that are sufficiently *close* to the stationary, but in general the chain never actually reaches it. Unless an analysis is provided that accounts for some initial distance to stationarity and mixing rate of the chain, I think it is a tad disingenuous to claim that the requirement for stationarity is a non-issue. I don't think not starting in stationarity is a condition that must be considered theoretically, but I **do** think at least one experiment exploring the effect of different burn ins in practical settings is required for the paper to tell a cohesive story.\n\nQUESTIONS:\n* The title of the submission in my opinion would benefit from being changed to being clearer about the specific setting considered. At present it masks the fact that the data are from a reversible chain, which could be quite niche.\n\n* Suggestion: It would be clearer to say from the outset (in the abstract even) that the points are assumed to come from a reversible process in stationary. \n\n* Why was the $\\sin^2$ error used in the analysis, as opposed to say, $L_p$ norms? Is it standard in analysis of such algorithms, if so, why? The paper could benefit from discussion of this aspect.\n\n* Issue: What are $\\mathcal{V}$ and $\\mathcal{M}$? These are not defined before their introduction (or even after, it seems), which is a serious problem.\n\n* The motivation for the work is indeed Metropolis-Hastings (for example) output, which justifies the reversible setting, but how much more difficult is the problem without it? Would one expect the result to change? A small discussion of this could possibly be beneficial.\n\n* Abstract, l2: \"principle\" should be \"principal\".\n\n* p2, l47: \"*the second eigenvalue in the magnitude of the transition matrix $P$*\" reads awkwardly and incorrectly (it sounds like the magnitude is taken wrt the matrix, not the eigenvalues).\n\n* Suggestion: Figure 1 would be easier to parse if the downsampling rate is mentioned in the legend.  \n\n* p5, l192: \"atleast\" should be \"at least\".\n\n* p7, equation after line 250 is significantly out of the page margins.\n\nLIMITATIONS:\nFor the most part yes, but additional clarity (as per other comments) regarding certain aspects (including the numerical experiments) would be beneficial for the paper to be up to NeurIPS standard in my opinion.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of streaming PCA, where the data points are revealed to the algorithm one at a time and the goal is to estimate the top eigenvector of the underlying covariance matrix.\nCrucially, the data points are sampled from a Markov chain and thus do not satisfy the typical assumption of being independent (they are still identically distributed as the Markov chain is stationary).\nThe paper analyzes the Oja's algorithm, which has shown a great empirical (and theoretical) success in streaming PCA with i.i.d. data. \nCrucially, the paper does not do any downsampling of the data, which is a common practice in the literature to curb the influence of dependence in the data.\n\n----\n# Update\nI thank the authors for their detailed responses. Based on their promise to include a nuanced discussion on the results in the final version of the paper (along with other presentation improvements suggested by the reviewers), I have now increased my score to 7 from 5.\n\nSTRENGTHS:\nThe paper studies an important problem and the paper's results are novel. In particular, they seem to improve over the de-facto approach of downsampling. I like the qualitative aspects of the paper's results.\n\nWEAKNESSES:\nIn terms of weaknesses, I have the following comments:\n\n1. (Writing) The paper's writing is sadly not up to the mark. In addition to the typos (some of which are listed below), theorem statements need to be simplified and the paper (especially Sections 4 and 5) need major re-writing. Even after spending a lot of time reading through the technical proof sketches in Sections 4 and 5, I was a bit clueless. \n\n2. (Gap-free bounds) There are no results for the gap-free setting, which is needed in several settings. I recommend adding them to strengthen the paper.\n\n3. (Dependence on failure probability) At several places, the paper claims to match the existing bounds in the literature for the i.i.d. setting. However, the dependence on the failure probability in the paper is much worse, polynomial as opposed to logarithmic in [15]. \n\n4. (Non-zero mean setting) Can the results be generalized to the zero mean setting? In the iid setting, one can simply take a pair of data and take their difference as the sample. Can one do the same trick here in the streaming setting (without compromising the mixing time)?\n\nQUESTIONS:\n# Questions\n\n1. Why are the error rates in the paper claimed to be sharp? Although it is reasonable to expect that a dependence on the mixing time appears, but does it have to be a multiplicative dependence? Are there lower bounds?\n\n2. Does the algorithm need to know the mixing time of the chain to declare victory? That is, Theorem 1 requires n to be larger than the mixing time. If yes, is it inherent?\n\n3. Isn't Corollary 1 immediate from the the i.i.d. result by using downsampling argument (without using Theorem 1)?\n\n# Other Comments\n\n1. Theorem 1 is too dense and difficult to parse. \n\n   i. Line 158 says that the gap $1- \\lambda_2(P)$ is inversely proportional to $\\tau_\\mathrm{mix}$. Then, why does Theorem 1 use both of them in the statement? It would be simple to use $\\tau_\\mathrm{mix}$.\n\n   ii. The notation with $\\alpha$ and $\\beta$ is convoluted. For the version in the main theorem, I suggest choosing a good enough value of $\\alpha$ and present a simplified choice of $\\beta$.\n\n2.  Line 194: Only the upper bounds obtained are sub-optimal. No lower bound has been shown.\n\n3. Section 5: What is the connection between the series of theorem statements.\n\n\n4. There are many typos and grammatical errors in the paper and many articles are missing. Several mathematical quantities are described after they are used. For example,\n    i. Line 31:  w.r.t.\n\n    ii. Line 31:  the true\n\n    iii. Line 36: v_1 is not defined.\n\n    iv. Line 75: What is meaning of 'stationary distribution of s\n\n    v. Assumption 1: What exactly is the last assumption is ambiguous.\n\n    vi. What norms are used in Assumptions 2 and 3?\n\n    vii. Line 139: WLOG\n\n    viii. Line 140: Why is it without loss of generality?\n\n    vix. Line 142: Braces around the equation are missing.\n\n    x. Section 2.1: Give specific references to the theorem statements in the book [18]. \n\n    xi. Line 181: Eq 1 should be Theorem 1.\n\n    xii.Line 182: Which remark in [15]?\n\n    xiii. Line 227 and 230: $\\alpha_{n,2}$ is not defined.\n\n    xiv. Line 228: $\\beta_{n,2}$ is not defined.\n\nLIMITATIONS:\nWhat are some instances where $\\mathcal{V}$ is much smaller than $\\mathcal{M}$. Using some simple calculations, I am getting similar expressions for them in the case of Gaussian data.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on a particular algorithm for PCA known as the Oja's algorithm. \nThis is an iterative algorithm that operates in a streaming manner. \nThe goal is to, when the data stream is Markovian, estimate the principal component of the covariance matrix of the unknown covariate vectors. \nThe main question that the authors addressed is to obtain sharp (up to multiplicative constant) rate guarantee for such an algorithm. \nThere are miscellaneous results in other parts of the paper that compare favorably to the main result of the paper. \nI found the most impressive aspect of this paper to be the fact that authors managed to shave the log factor in the rate (compared to the down-sampled version commonly used in Markovian setting).\n\nSTRENGTHS:\nThe result of this paper seems rather satisfactory and tells a relatively complete story. \nThe main text is somewhat dense but it's never getting obscure. \nContributions, techniques to achieve them and difficulties for doing so are clearly explained.\n\nWEAKNESSES:\nI don't see obvious weaknesses. \nSo I'll just propose a few minor suggestions. \n\nI personally found it useful to keep the following binary Markov chain in mind when reading the results and proof sketches: the states are -1, +1 and the transition kernel is given by a binary symmetric channel with flip probability delta. \nThen the spectral gap is delta (perhaps up to constant) and the mixing time is essentially 1/delta. \nI encourage the authors to use this as a running example in the proof sketch, though I don't strongly insist on this. \n\nIf I understand correctly, the Markov chain does not have to be finite-state right?\nIf so, this is perhaps worth commenting on, if it has not been done yet.\n\nQUESTIONS:\n1. The sin^2 guarantees in various theorem statements are somewhat heavy. Does it make sense to the authors to state a more compact bound (say using big-O notation) in the main text and defer the exact bound to appendices? On the other hand, I do understand that the authors would like to present completely nonasymptotic bounds...\n\n2. I did not check the proof carefully, but my impression upon scanning the appendices is that the proof seems to rely heavily on the simplicity of the form of the Oja's iteration. \nFor instance, there're intimate relation between this work and a line of work on concentration for random matrix products. \nGoing beyond Oja's iteration, it possible to extrapolate the proof techniques to other iterations where there are nonlinearities?\nE.g., in generalized power iteration, approximate message passing and so on, it's important to pick judicious nontrivial nonlinearities. \nOf course, not all of these algorithms are applicable in the Markovian setting, but I'm sure the authors are aware of applicable examples. \n\n3. The authors have already discussed the down-sampling trick in details. I would like to add a few more examples where there's loss of log factors using down-sampling. Bresler and his students have a line of work on inference in the presence of temporal correlation, notably this one https://arxiv.org/abs/2006.08916 in the context of linear regression. The idea there is precisely down-sampling and they showed down-sampling is optimal *up to log factors*. Clearly the authors of this submission is not satisfied with log factors. But it's still interesting to see similar phenomenon being recurrently observed. \nAnother \"example\" is in the setting of parameter estimation from Gaussian mixtures where the mixing coefficients form a binary Markov chain; see https://arxiv.org/abs/2206.02455. \nOver there, curiously, one cannot waste samples. Instead, one takes coherent combination within blocks of size being roughly the mixing time of the chain.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

comments: SUMMARY:
This paper analyzes the performance of Oja'a algorithm for streaming PCA when the samples come from an irreducible, aperiodic, and reversible Markov chain. In this work the authors avoid using downsampling to reduce dependency, and their result improves on that proposed in [3] by a poly-logarithmic multiplicative factor.

STRENGTHS:
This paper gives a near optimal upper bound on the estimation error of Oja's algorithm for streaming PCA with Markovian data. Most past results consider iid data, while in many practical applications the data are dependent. Their result improve upon previous error rate bounds, and nearly matches an offline algorithm. Careful comparisons of the error rates are also provided in the paper. Leveraging the mixing properties of Markov chains, they present novel analysis that might find applications in other online problems.

WEAKNESSES:
It seems that knowledge of the transition matrix is required to produce step sizes. In practice this must be estimated, it will be nice to include this as a part of the algorithm.

QUESTIONS:
1. What is C in line 128? C also appear as a constant later. 
2. $\pi$ is the stationary distribution of $X$ or $s$? In line 46 $\pi$ seems to be the stationary distribution for $X$, while in section 2 $\pi$ seems to be the stationary distribution for $s$. 
3. Assumption is not satisfied in several basic settings for example $X_i$ are Gaussian, which is a little disappointing. Can this assumption be relaxed using some additional arguments, for example, truncation?

LIMITATIONS:
Yes. Societal impact not applicable.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07