SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The leading term of Theorem 1 is worse by a factor of  $\frac{1}{1 - |\lambda_2(P)|}$ . Further, it has an additive lower order term  $O\left(\frac{\log^2(n)}{n^2}\right)$  due to the covariance between data-points in the Markovian case.
Corollary 1. (Downsampled Oja's algorithm) Fix a  $\delta \in (0,1)$ . If Oja's algorithm is applied on the downsampled data-stream with every  $k^{th}$  data-point, where  $k \coloneqq \tau_{\text{mix}}(\eta_n^2)$  then under the conditions of Theorem 1 with appropriately modified  $\alpha$  and  $\beta$ , the output  $w_n$  satisfies,
$$
\begin{array}{l} 1 - \left(w _ {n} ^ {T} v _ {1}\right) ^ {2} \leq \\ \frac {C \log \left(\frac {1}{\delta}\right)}{\delta^ {2}} \left[ d \left(\frac {2 \beta \tau_ {m i x} \log (n)}{n}\right) ^ {2 \alpha} + \frac {C _ {1} \mathcal {V} \tau_ {m i x}}{\left(\lambda_ {1} - \lambda_ {2}\right) ^ {2}} \frac {\log (n)}{n} + \frac {C _ {2} \mathcal {M} \left(\mathcal {M} + \lambda_ {1}\right) ^ {2}}{\left(\lambda_ {1} - \lambda_ {2}\right) ^ {3}} \frac {\log^ {2} (n) \tau_ {m i x} \left(\eta_ {n} ^ {2}\right) ^ {2}}{n ^ {2}} \right] \\ \end{array}
$$
with probability at least  $(1 - \delta)$ . Here  $C$  is an absolute constant and  $C_1 \coloneqq \frac{30\alpha^2}{2\alpha - 1}$ ,  $C_2 \coloneqq \frac{35\alpha^3}{\alpha - 1}$ .
Remark 5. Data downsampling to reduce dependence amongst samples has been suggested in recent work [27, 23, 3]. In Corollary 1, we establish that the bound on the rate obtained is sub-optimal compared to Theorem 1 by a log  $(n)$  factor. We prove this by a simple yet elegant observation: the downsampled data stream can be considered to be drawn from a Markov chain with transition kernel  $P^k (.,.)$  since each data-point is  $k$  steps away from the previous one. For sufficiently large  $k$ , this implies that the mixing time of this chain is  $\Theta (1)$ . These new parameters are used to select the modified values of  $\alpha, \beta$  according to Lemma S.11 in the Supplement.
The proof of Theorem 1 follows the same general recipe as in [16] for obtaining a bound on the  $\sin^2$  error. However, the original proof techniques heavily rely on the IID setting. We carry out a refined analysis for each step under the Markovian data model by a careful control of error terms arising out of dependence. The first step involves obtaining a high-probability bound on the  $\sin^2$  error, by noting that Oja's algorithm on  $n$  data-points can be viewed as a single iteration of the power method on  $B_{n}$ . Therefore, fixing a  $\delta \in (0,1)$  using Lemma 3.1 from [16], we have with probability at least  $(1 - \delta)$ ,
$$
\sin^ {2} \left(w _ {n}, v _ {1}\right) \leq \frac {C \log \left(\frac {1}{\delta}\right)}{\delta} \frac {\operatorname {T r} \left(V _ {\perp} ^ {T} B _ {n} B _ {n} ^ {T} V _ {\perp}\right)}{v _ {1} ^ {T} B _ {n} B _ {n} ^ {T} v _ {1}}, \tag {6}
$$
where  $C$  is an absolute constant. The numerator is bounded by first bounding its expectation (see Theorem 3) and then using Markov's inequality. To bound the denominator, similar to [16], we will use Chebyshev's inequality. Theorem 4 provides a lower bound for the expectation  $\mathbb{E}\left[v_1^T B_nB_n^T v_1\right]$ . Chebyshev's inequality also requires upper-bounding the variance of  $\mathbb{E}\left[v_1^T B_nB_n^T v_1\right]$ , which requires us to bound  $\mathbb{E}\left[\left(v_1^T B_nB_n^T v_1\right)^2\right]$  (see Theorem 5).
4 Main Technical Tools
In this section, we provide a sketch of the main arguments used in our proof.
Warm-up with downsampled Oja's algorithm: We start with the simple downsampled Oja's algorithm to build intuition. Here, one applies Oja's update rule (Eq 1) to every  $k^{th}$  data-point, for a suitably chosen  $k$ . For  $k = \lceil L\tau_{\mathrm{mix}}\log n\rceil$ , the total variation distance between any consecutive data-points in the downsampled data stream is  $O(n^{-L})$ . As we show in Corollary 1, the error of this algorithm is similar to the error of Oja's algorithm applied to  $n / k$  data-points in the IID setting, i.e.,  $O(\mathcal{V}\tau_{\mathrm{mix}}\log n / n)$ .
We will take  $\mathbb{E}\left[v_1^T B_nB_n^T v_1\right]$  as an example. Let us introduce some notation.
$$
B _ {j, i} := \left(I + \eta_ {j} X _ {j} X _ {j} ^ {T}\right) \left(I + \eta_ {j - 1} X _ {j - 1} X _ {j - 1} ^ {T}\right) \dots \left(I + \eta_ {i} X _ {i} X _ {i} ^ {T}\right) \tag {7}
$$
We peel this quantity one matrix at a time from the inside. Note that for a reversible Markov chain, standard results imply (see Lemma 1) that the mixing conditions apply to the conditional distribution of a state given another state  $k$  steps in the "future" (see Supplement Section S.3 for a proof). Recall  $d_{\mathrm{mix}}(k)$  from Section 2.1.
Lemma 1. Under Assumption 1,  $\frac{1}{2}\sup_{t\in \Omega}\sum_s|\mathbb{P}(Z_t = s|Z_{t + k} = t) - \pi (s)| = d_{mix}(k)$ .
It will be helpful to explain our analysis by comparing it with the IID setting. For this reason, we will use  $\mathbb{E}_{\mathrm{IID}}[.]$  to denote the expectation under the IID data model. Define  $\alpha_{n,i} \coloneqq \mathbb{E}\left[v_1^T B_{n,i}B_{n,i}^T v_1\right], i \in [n]$ . Then we have,
$$
\begin{array}{l} \alpha_ {n, 1} = \mathbb {E} \left[ v _ {1} ^ {T} B _ {n, 2} \left(I + \eta_ {1} \Sigma + \eta_ {1} \left(X _ {1} X _ {1} ^ {T} - \Sigma\right)\right) \left(I + \eta_ {1} \Sigma + \eta_ {1} \left(X _ {1} X _ {1} ^ {T} - \Sigma\right)\right) ^ {T} B _ {n, 2} ^ {T} v _ {1} \right] \\ = \mathbb {E} \left[ v _ {1} ^ {T} B _ {n, 2} \left(I + \eta_ {1} \Sigma\right) ^ {2} B _ {n, 2} ^ {T} v _ {1} \right] + 2 \eta_ {1} T _ {1} + \eta_ {1} ^ {2} T _ {2}, \tag {8} \\ \end{array}
$$
where the first term is smaller than  $(1 + \eta_1\lambda_1)^2\alpha_{n,2}$ . We define  $T_{1}$  and  $T_{2}$  as follows.  $T_{1} := \mathbb{E}\left[v_{1}^{T}B_{n,2}\left(I + \eta_{1}\Sigma\right)\left(X_{1}X_{1}^{T} - \Sigma\right)B_{n,2}^{T}v_{1}\right]$ , and  $T_{2} := \mathbb{E}\left[v_{1}^{T}B_{n,2}\left(X_{1}X_{1}^{T} - \Sigma\right)^{2}B_{n,2}^{T}v_{1}\right]$ .
For the IID setting, the second term is zero, and the third term can be bounded as follows:
$$
\begin{array}{l} \mathbb {E} _ {\text {I I D}} \left[ v _ {1} ^ {T} B _ {n, 2} \left(X _ {1} X _ {1} ^ {T} - \Sigma\right) ^ {2} B _ {n, 2} ^ {T} v _ {1} \right] = \mathbb {E} _ {\text {I I D}} \left[ v _ {1} ^ {T} B _ {n, 2} \mathbb {E} \left[ \left(X _ {1} X _ {1} ^ {T} - \Sigma\right) ^ {2} \right] B _ {n, 2} ^ {T} v _ {1} \right] \\ \leq \mathcal {V} \mathbb {E} _ {\mathrm {I D}} \left[ v _ {1} ^ {T} B _ {n, 2} B _ {n, 2} ^ {T} v _ {1} \right] \\ \end{array}
$$
Let us denote the IID version of  $\alpha_{n,i}$  by  $\alpha_{n,i}^{\mathrm{IID}}\coloneqq \mathbb{E}_{\mathrm{IID}}[v_1^T B_{n,i}B_{n,i}^T v_1]$ . The final recursion for the IID case becomes:  $\alpha_{n,1}^{\mathrm{IID}}\leq (1 + 2\eta_1\lambda_1 + \eta_1^2 (\lambda_1^2 +\mathcal{V}))\alpha_{n,1}^{\mathrm{IID}}$ . So, for our Markovian data model, the hope is that the cross term  $T_{1}$  (which has a multiplicative factor of  $\eta_{1}$ ) is  $O(\eta_{1})$  and  $T_{2}$  is  $O(\eta_1^2)$ . We will start with the  $T_{1}$  term, which is zero in the IID setting.
Figure 2: If we could replace the intermediate products (white matrices) by  $I$ , the conditional expectation of the noise matrix  $X_{1}X_{1}^{T} - \Sigma$  conditioned on the gray matrices would be nearly zero.
We will show that  $T_{1}$ , while not zero like the IID case, is still sufficiently small. Intuitively, if we could replace the  $k$  matrices between  $X_{1}X_{1}^{T} - \Sigma$  and  $B_{n,2 + k}$  for some suitably large integer  $k$  by identity (see Figure 2), then using (reverse) mixing properties of the Markov chain, we could argue using Lemma 1 that  $\mathbb{E}[X_1X_1^T - \Sigma | s_{1 + k}, \ldots, s_n]$  is very close to zero (see Figure 2). The following lemma formally bounds the deviation of the length-  $k$  matrix product from identity.
Lemma 2. Let Assumption 3 hold. If  $\forall i\in [n],\eta_i k_i(\mathcal{M} + \lambda_1)\leq \epsilon ,\epsilon \in (0,1)$  and  $\eta_{i}$  forms a non-increasing sequence then  $\forall m\leq n - k_n$
$$
\left\| B _ {m + k _ {m} - 1, m} - I \right\| _ {2} \leq (1 + \epsilon) k _ {m} \eta_ {m} (\mathcal {M} + \lambda_ {1}) a n d \tag {9}
$$
$$
\left\| B _ {m + k _ {m} - 1, m} - I - \sum_ {t = m} ^ {m + k _ {m} - 1} \eta_ {t} X _ {t} X _ {t} ^ {T} \right\| _ {2} \leq k _ {m} ^ {2} \eta_ {m} ^ {2} \left(\mathcal {M} + \lambda_ {1}\right) ^ {2} \tag {10}
$$
Lemma 2 bounds the norm of the matrix product  $B_{t + k_t - 1,t}$  at two levels. The first result provides a coarse bound, approximating linear and higher-order terms. The second result provides a finer bound, preserving the linear term and approximating quadratic and higher-order terms. The proofs involve a straightforward combinatorial expansion of  $B_{t + k_t - 1,t}$  and are deferred to the Supplement Section S.3.
Approximating  $\prod_{i=2}^{k+1}(I + \eta_i X_i X_i^T)$  requires  $\eta_1 k$  to be small. Since this is a recursive argument, we would need  $\eta_i k$  to be small for  $i = 1, \ldots, n$ , which is satisfied by the strong condition  $\eta_1 k$  is small. To obtain a tight analysis, we choose  $k$  adaptively. We set  $k_i = \tau_{\mathrm{mix}}(\eta_i^2)$  (see definition in Eq 3).
As we will show in detail in the Supplement, Lemma 2 Eq 10 along with the adaptive choice of  $k_{i}$  gives us a sharp error bound. Using it, we can bound  $T_{1}$  (see Eq 8) as:
$$
\begin{array}{l} T _ {1} \leq \sum_ {j = 2} ^ {k + 1} \eta_ {j} \mathbb {E} \left[ v _ {1} ^ {T} B _ {n, k + 2} \underbrace {\mathbb {E} \left[ \left(X _ {j} X _ {j} ^ {T}\right) (I + \eta_ {1} \Sigma) \left(X _ {1} X _ {1} - \Sigma\right) \mid X _ {k + 2} , . . . , X _ {n} \right]} _ {T _ {1, j}} B _ {n, k + 2} ^ {T} v _ {1} \right] \\ + O (\eta_ {1} ^ {2} k _ {1} ^ {2}) \alpha_ {n, k + 2} \\ \end{array}
$$
Naively bounding the  $T_{1,j}$  term by  $O(1)$  leads to the same rate as downsampled Oja's algorithm.
In the following lemma, we will establish that, indeed,  $T_{1,j}$  has a much smaller norm. The novelty of our bound is not just in using the mixing properties of the Markov chain but also in teasing out the variance parameter  $\mathcal{V}$ . We will state the lemma, in a slightly more general form as -
Lemma 3. Under Assumptions 1, 2 and 3, for  $i < j \leq i + k_i$ ,
$$
\left. \left\| \mathbb {E} \left[ \left(X _ {i} X _ {i} ^ {T} - \Sigma\right) S X _ {j} X _ {j} ^ {T} | s _ {i + k _ {i}}, \dots s _ {n} \right] \right\| _ {2} \leq \left(| \lambda_ {2} (P) | ^ {j - i} \mathcal {V} + 8 \eta_ {i} ^ {2} \mathcal {M} (\mathcal {M} + \lambda_ {1})\right) \| S \| _ {2} \right.
$$
where  $k_{i}$  is as defined in Lemma S.11 and  $S$  is a constant symmetric positive semi-definite matrix.
Lemma 3 bounds the norm of the covariance between matrices  $\left(X_{i}X_{i}^{T} - \Sigma\right)S$  and  $X_{j}X_{j}^{T}$ . In particular, this implies that the norm of  $T_{1,j}$  decays as  $|\lambda_2(P)|^{j-1}$ . The proof uses a spectral argument that replaces a coarse approximation by a sum of  $k_{i}O(1)$  terms to sum of  $k$  exponentially decaying terms, thereby removing the dependence on  $k_{i}$ , which can be as large as  $\log(n)$ . The proof is deferred to the Supplement Section S.4.
Let  $\{c_1, c_2, c_3, c_4\}$  be positive constants for ease of notation. Coming back to Eq 8, we can bound  $T_1$  as follows:  $T_1 \leq \alpha_{n,k+2} \left( \eta_1 \frac{c_1 |\lambda_2(P)| \mathcal{V}}{1 - |\lambda_2(P)|} + c_2 \eta_1^2 k_1^2 \right)$ . A similar argument can be applied to bound  $T_2$  as:  $T_2 \leq \alpha_{n,k+2} (\mathcal{V} + c_3 \eta_1 k_1^2)$ . Putting everything together in 8, we have
$$
\alpha_ {n, 1} \leq \underbrace {\left((1 + \eta_ {1} \lambda_ {1}) ^ {2} + \mathcal {V}\right) \alpha_ {n , 2}} _ {\text {R e c u r s i o n f o r I I D s e t t i n g}} + \underbrace {\left(\frac {c _ {1} | \lambda_ {2} (P) |}{1 - | \lambda_ {2} (P) |}\right) \mathcal {V} \eta_ {1} ^ {2} \alpha_ {n , k + 2}} _ {\text {E r r o r d u e t o M a r k o v i a n d e p e n d e n c e}} + \underbrace {c _ {4} \eta_ {1} ^ {3} k _ {1} ^ {2} \alpha_ {n , k + 2}} _ {\text {E r r o r d u e t o a p p r o x i m a t i o n o f m a t r i x p r o d u c t i o n}}
$$
Recursing on this inequality gives us our bound on  $\mathbb{E}\left[v_1^T B_n B_n^T v_1\right]$  (Theorem 2). We are now ready to present all our accompanying theorems.
5 Intermediate Theorems for Convergence Analysis
In this section, we present our accompanying theorems which are used to obtain the main result in Theorem 1. But before doing so, we will need to establish some notation. Let  $k_{i} \coloneqq \tau_{\mathrm{mix}}(\eta_{i}^{2})$ , and the step-sizes be set as  $\eta_{i} \coloneqq \frac{\alpha}{(\lambda_{1} - \lambda_{2})(\beta + i)}$  with  $\alpha, \beta$  as defined in Theorem 1. Let  $\epsilon \coloneqq \frac{1}{100}$ . As shown in Lemma S.11 in Supplement Section S.3 our choice of step-sizes satisfy,  $\forall i \in [n]$ ,
C.1  $\eta_{i}k_{i}\left(\mathcal{M} + \lambda_{1}\right)\leq \epsilon$
C.2 (Slow decay)  $\eta_{i}\leq \eta_{i - k_{i}}\leq (1 + 2\epsilon)\eta_{i}\leq 2\eta_{i}$
Further, we define scalar variables -
$$
\begin{array}{l} r := 2 \left(1 + \epsilon\right) k _ {n} \eta_ {n} \left(\mathcal {M} + \lambda_ {1}\right), \qquad \zeta_ {k, t} := 4 0 k _ {t + 1} \left(\mathcal {M} + \lambda_ {1}\right) ^ {2} \\ \psi_ {k, t} := 6 \mathcal {M} \left[ 1 + 3 k _ {t + 1} ^ {2} \left(\mathcal {M} + \lambda_ {1}\right) ^ {2} \right], \quad \mathcal {V} ^ {\prime} := \frac {1 + (3 + 4 \epsilon) | \lambda_ {2} (P) |}{1 - | \lambda_ {2} (P) |} \mathcal {V} \tag {11} \\ \end{array}
$$
and recall the definitions of  $B_{t}$  and  $B_{j,i}$  in Eqs 2 and 7, respectively. We are now ready to present the theoretical results needed to prove our main result. For simplicity of notation, we present versions of the results by using  $\eta_{i} := \frac{\alpha}{(\lambda_{1} - \lambda_{2})(\beta + i)}$  with  $\alpha, \beta$  as defined in Theorem 1. However, these theorems are in fact valid under more general step-size schedules. We state and prove the more general versions in the Supplement Section S.4.
Theorem 2. Under Assumptions 1, 2 and 3, for all  $n > k_n$ , and  $\eta_i$  satisfying C.1 and C.2, we have,
$$
\mathbb {E} \left[ v _ {1} ^ {T} B _ {n} B _ {n} ^ {T} v _ {1} \right] \leq (1 + r) ^ {2} \exp \left(\sum_ {t = 1} ^ {n - k _ {n}} \left(2 \eta_ {t} \lambda_ {1} + \eta_ {t} ^ {2} \left(\mathcal {V} ^ {\prime} + \lambda_ {1} ^ {2}\right) + \eta_ {t} ^ {3} \psi_ {k, t}\right)\right)
$$
The three primary differences with the IID case are a) the  $(1 + r)^{2}$  term, which arises since the recursion sketched in Section 4 leaves out the last  $k_{n}$  terms which are bounded by  $(1 + r)^{2}$ ; (b) the new factor of  $\frac{1}{1 - |\lambda_2(P)|}$  with  $\nu$  due to the Markovian dependence between terms; and c) the extra lower order term  $\eta_t^3\psi_{k,t}$  arising from the use of Lemmas 2 and 3.
Theorem 3. Let  $u \coloneqq \min \{t : t \in [n], t - k_t \geq 0\}$ . Under Assumptions 1, 2 and 3, for all  $n > u$  and  $\eta_i$  satisfying C.1 and C.2,
$$
\begin{array}{l} \mathbb {E} \left[ \operatorname {T r} \left(V _ {\perp} ^ {T} B _ {n} B _ {n} ^ {T} V _ {\perp}\right) \right] \leq (1 + 5 \epsilon) \exp \left(\sum_ {t = u + 1} ^ {n} 2 \eta_ {t} \lambda_ {2} + \eta_ {t - k _ {t}} ^ {2} \left(\mathcal {V} ^ {\prime} + \lambda_ {1} ^ {2}\right) + \eta_ {t - k _ {t}} ^ {3} \psi_ {k, t}\right) \\ \times \left(d + \sum_ {t = u + 1} ^ {n} \left(\mathcal {V} ^ {\prime} + \eta_ {t} \psi_ {k, t}\right) C _ {k, t} ^ {\prime} \eta_ {t - k _ {t}} ^ {2} \exp \left(\sum_ {i = u + 1} ^ {t} 2 \eta_ {i} \left(\lambda_ {1} - \lambda_ {2}\right)\right)\right) \\ \end{array}
$$
where  $C_{k,t}^{\prime} \coloneqq \left(1 + \frac{\delta}{200}\right)\exp \left(2\lambda_1\sum_{i = 1}^{u}\eta_j\right)$ .
Here, the difference is mainly in the new variable  $u$ , arising since the recursion stops at  $u$ , not 1.  $(1 + 5\epsilon)$  represents the approximation of the first  $u$  terms.
Theorem 4. Under Assumptions 1, 2 and 3, for all  $n > k_n$ ,  $\eta_i$  satisfying C.1 and C.2, and  $s := 2r + \frac{\delta}{1000}$ , we have,
$$
\mathbb {E} \left[ v _ {1} ^ {T} B _ {n} B _ {n} ^ {T} v _ {1} \right] \geq (1 - s) \exp \left(\sum_ {t = 1} ^ {n - k _ {n}} 2 \eta_ {t} \lambda_ {1} - \sum_ {t = 1} ^ {n - k _ {n}} 4 \eta_ {t} ^ {2} \lambda_ {1} ^ {2}\right)
$$
This differs from its IID counterpart by a multiplicative factor of  $(1 - s)$  for the same reason as before, which also makes the sums go up to  $(n - k_n)$  instead of  $n$ . Note that for sufficiently large  $n$  (Lemma S.12),  $r = O\left(\frac{\log(n)}{n}\right)$  is very small and  $\delta \in (0, 1)$ . Therefore,  $(1 - s) \approx 1$  as large  $n$ .
Theorem 5. Under Assumptions 1, 2 and 3, for all  $n > k_n$ , and  $\eta_i$  satisfying C.1 and C.2, we have,
$$
\mathbb {E} \left[ \left(v _ {1} ^ {T} B _ {n} B _ {n} ^ {T} v _ {1}\right) ^ {2} \right] \leq (1 + r) ^ {4} \exp \left(\sum_ {t = 1} ^ {n - k _ {n}} 4 \eta_ {t} \lambda_ {1} + \sum_ {t = 1} ^ {n - k _ {n}} \eta_ {t} ^ {2} \zeta_ {k, t}\right)
$$
The differences are similar to the last theorems involving  $v_{1}$ . Surprisingly, for this, the coarse approximation suffices, leading to an absence of the  $\mathcal{V}$  term in the bound. Having established these results, the final step is to substitute them into Eq 6 and follow the proof recipe described earlier. This requires significant calculations and is deferred to the Supplement Section S.5.
In this section, we present some simple experiments to validate our theoretical results. For more detailed experiments, see the Supplement Section S.6. We design a Markov chain with  $|\Omega| = 10$  states, where the transition matrix entries  $P_{ij}$  equal  $\rho / (|\Omega| - 1)$  for  $i \neq j$  and  $1 - \rho$  for  $i = j$ . Smaller values of  $\rho$  lead to larger mixing times. It can be verified that the stationary distribution  $\pi = \mathcal{U}(\Omega)$  is uniform over the state-space and  $|\lambda_2(P)| \approx (1 - \rho)$ . We set  $\rho = 0.2$  for Figures 1 and 3a, and vary it in Figure 3b. Each point in the plot is averaged over 20 random runs over different Markov chains, datasets, and initialization.
(a) Variation of  $\sin^2$  error with  $\beta$
(b) Variation of  $\sin^2$  error with  $|\lambda_2(P)|$
Figure 3: X axis represents the sample size, and Y axis represents the  $\sin^2$  error.
Each state  $s \in \Omega$  is associated with  $D(s) \coloneqq \operatorname{Bernoulli}(p_s)$  distribution. We set  $d = 1000$  and select  $p_s \sim \mathcal{U}(0,0.05)$  at the start of each random run. The covariance matrix,  $\Sigma_s$ , for each state is set as  $\Sigma_s(i,j) = \exp(-|i-j|c_s)\sigma_i\sigma_j$  where  $c_s \coloneqq 1 + 9\left(\frac{s-1}{|\Omega|-1}\right)$ ,  $\sigma_i \coloneqq 5i^{-\beta}$ . We start with the stationary distribution  $\pi$ , and for each state  $s_i$ , we draw IID samples  $Z_i \sim D(s_i)$ . We standardize  $Z_i$  such that all components have zero mean and unit variance under the state distribution,  $D(s_i)$ . We then generate the sample data-point for PCA as  $X_i = \Sigma_i^{\frac{1}{2}}Z_i$ . By construction,  $\mathbb{E}_{D(s_i)}[X_iX_i^T] = \Sigma_i$  and  $\mathbb{E}[X_i] = 0^d$ . The step sizes for Oja's algorithm are set as  $\eta_i = \frac{\alpha}{(\beta+i)(\lambda_1-\lambda_2)}$  for  $\alpha = 5, \beta = \frac{5}{1 - |\lambda_2(P)|}$ . For the downsampled variant, every  $10^{th}$  data-point is considered, and  $\beta$  is accordingly divided by 10. For the offline algorithm, we recompute the leading eigenvector of the sample covariance matrix of data-points seen so far.
Figure 1 compares the performance of different algorithms for the Bernoulli distribution. Here, we are checking if the results obtained in Theorem 1, Proposition 1, and Corollary 1 are reflected in the experiments. The experimental results demonstrate that Oja's algorithm performs significantly better than the downsampled version, consistent with the theoretical results. It also shows that Oja's algorithm performs similarly to the offline algorithm, which is also confirmed by our theoretical results and that of [28]. Figure 3a compares the performance of Oja's algorithm for different covariance matrices. Smaller values of  $\beta$  decrease the eigengap  $\lambda_{1} - \lambda_{2}$ , and hence lead to a slower convergence. Figure 3b confirms that smaller values of  $\rho$  (larger values of  $|\lambda_2(P)|$ ) also worsen the rate, which matches with our theoretical results.
We have considered the problem of streaming PCA for Markovian data, which has implications in various settings like decentralized optimization, reinforcement learning, etc. The analysis of streaming algorithms in such settings has seen a renewed surge of interest in recent years. However, the dependence between data-points makes it difficult to obtain sharp bounds. We provide, to our knowledge, the first near-optimal bound for obtaining the first principal component from a Markovian data stream that breaks the logarithmic barrier present in the analysis done for downsampled data. We believe that the theoretical tools that we have developed in this paper would enable one to obtain sharp bounds for other dependent data settings, learning top  $k$  principal components, and online inference algorithms with updates involving products of matrices.
We gratefully acknowledge NSF grants 2217069, 2019844, and DMS 2109155. We are also grateful to Rachel Ward, Bobby Shi, and Soumendu Sundar Mukherjee for useful discussions and to the anonymous reviewers for their valuable feedback.
[38] Puyudi Yang, Cho-Jui Hsieh, and Jane-Ling Wang. History pca: A new algorithm for streaming pca. arXiv preprint arXiv:1802.05447, 2018.
[39] Ingvar Ziemann and Stephen Tu. Learning with little mixing. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 4626-4637. Curran Associates, Inc., 2022.