SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Stefan Stojanovic
stesto@kth.se
Yassir Jedra
jedra@kth.se
We study matrix estimation problems arising in reinforcement learning (RL) with low-rank structure. In low-rank bandits, the matrix to be recovered specifies the expected arm rewards, and for low-rank Markov Decision Processes (MDPs), it may for example characterize the transition kernel of the MDP. In both cases, each entry of the matrix carries important information, and we seek estimation methods with low entry-wise error. Importantly, these methods further need to accommodate for inherent correlations in the available data (e.g. for MDPs, the data consists of system trajectories). We investigate the performance of simple spectral-based matrix estimation approaches: we show that they efficiently recover the singular subspaces of the matrix and exhibit nearly-minimal entry-wise error. These new results on low-rank matrix estimation make it possible to devise reinforcement learning algorithms that fully exploit the underlying low-rank structure. We provide two examples of such algorithms: a regret minimization algorithm for low-rank bandit problems, and a best policy identification algorithm for reward-free RL in low-rank MDPs. Both algorithms yield state-of-the-art performance guarantees.
Learning succinct representations of the reward function or of the system state dynamics in bandit and RL problems is empirically known to significantly accelerate the search for efficient policies [38, 55, 13]. It also comes with interesting theoretical challenges. The design of algorithms learning and leveraging such representations and with provable performance guarantees has attracted considerable attention recently, but remains largely open. In particular, significant efforts have been made towards such design when the representation relies on a low-rank structure. In bandits, assuming such a structure means that the arm-to-reward function can be characterized by a low-rank matrix [37, 32, 7, 27]. In MDPs, it implies that the reward function, the  $Q$ -function or the transition kernels are represented by low-rank matrices [56, 4, 46, 60, 49]. In turn, the performance of algorithms exploiting low-rank structures is mainly determined by the accuracy with which we are able to estimate these matrices.
In this paper, we study matrix estimation problems arising in low-rank bandit and RL problems. Two major challenges are associated with these problems. (i) The individual entries of the matrix carry important operational meanings (e.g. in bandits, an entry could correspond to the average reward of an arm), and we seek estimation methods with low entry-wise error. Such requirement calls for a
fine-grained analysis, typically much more involved than that needed to only upper bound the spectral or Frobenius norm of the estimation error [22, 21, 12, 2, 54, 15, 53, 47]. (ii) Our estimation methods should further accommodate for inherent correlations in the available data (e.g., in MDPs, we have access to system trajectories, and the data is hence Markovian). We show that, essentially, spectral methods successfully deal with these challenges.
Contributions. 1) We introduce three matrix estimation problems. The first arises in low-rank bandits. The second corresponds to scenarios in RL where the learner wishes to estimate the (low-rank) transition kernel of a Markov chain and to this aim, has access to a generative model. The last problem is similar but assumes that the learner has access to system trajectories only, a setting referred to as the forward model in the RL literature. For all problems, we establish strong performance guarantees for simple spectral-based estimation approaches: these efficiently recover the singular subspaces of the matrix and exhibit nearly-minimal entry-wise error. To prove these results, we develop and combine involved leave-one-out arguments and Poisson approximation techniques (to handle the correlations in the data).
2) We apply the results obtained for our first matrix estimation problem to devise an efficient regret-minimization algorithm for low-rank bandits. We prove that the algorithm enjoys finite-time performance guarantees, with a regret at most roughly scaling as  $(m + n)\log^3 (T)\bar{\Delta} /\Delta_{\mathrm{min}}^2$  where  $(m,n)$  are the reward matrix dimensions,  $T$  is the time horizon,  $\bar{\Delta}$  is the average of the reward gaps between the best arm and all other arms, and  $\Delta_{\mathrm{min}}$  is the minimum of these gaps.
3) Finally, we present an algorithm for best policy identification in low-rank MDPs in the reward-free setting. The results obtained for the second and last matrix estimation problems imply that our algorithm learns an  $\epsilon$ -optimal policy for any reward function using only a number of samples scaling as  $O(nA / \epsilon^2)$  up to logarithmic factors, where  $n$  and  $A$  denote the number of states and actions, respectively. This sample complexity is mini-max optimal [28], and illustrates the gain achieved by leveraging the low-rank structure (without this structure, the sample complexity would be  $\Omega(n^2 A / \epsilon^2)$ ).
Notation. For any matrix  $A \in \mathbb{R}^{m \times n}$ ,  $A_{i,j}$  (resp.  $A_{\cdot,j}$ ) denotes its  $i$ -th row (resp. its  $j$ -th column),  $A_{\min} = \min_{(i,j)} A_{i,j}$  and  $A_{\max} = \max_{(i,j)} A_{i,j}$ . We consider the following norms for matrices:  $\|A\|$  denotes the spectral norm,  $\|A\|_{1 \to \infty} = \max_{i \in [m]} \|A_{i,\cdot}\|_1$ ,  $\|A\|_{2 \to \infty} = \max_{i \in [m]} \|A_{i,\cdot}\|_2$ , and finally  $\|A\|_{\infty} = \max_{(i,j) \in [m] \times [n]} |A_{i,j}|$ . If the SVD of  $A$  is  $U\Sigma V^{\top}$ , we denote by  $\mathrm{sgn}(A) = UV^{\top}$  the matrix sign function of  $A$  (see Definition 4.1 in [14]).  $\mathcal{O}^{r \times r}$  denotes the set of  $(r \times r)$  real orthogonal matrices. For any finite set  $S$ , let  $\mathcal{P}(S)$  be the set of distributions over  $S$ . The notation  $a(n,m,T) \lesssim b(n,m,T)$  (resp.  $a(n,m,T) = \Theta(b(n,m,T))$ ) means that there exists a universal constant  $C > 0$  (resp.  $c, C > 0$ ) such that  $a(n,m,T) \leq C b(n,m,T)$  (resp.  $cb(n,m,T) \leq a(n,m,T) \leq C b(n,m,T)$ ) for all  $n, m, T$ . Finally, we use  $a \wedge b = \min(a,b)$  and  $a \vee b = \max(a,b)$ .
2 Models and Objectives
Let  $M \in \mathbb{R}^{m \times n}$  be an unknown rank  $r$  matrix that we wish to estimate from  $T$  noisy observations of its entries. We consider matrices arising in two types of learning problems with low-rank structure, namely low-rank bandits and RL. The SVD of  $M$  is  $U\Sigma V^{\top}$  where the matrices  $U \in \mathbb{R}^{m \times r}$  and  $V \in \mathbb{R}^{n \times r}$  contain the left and right singular vectors of  $M$ , respectively, and  $\Sigma = \mathrm{diag}(\sigma_1, \dots, \sigma_r)$ . We assume without loss of generality that the singular values have been ordered, i.e.,  $\sigma_1 \geq \dots \geq \sigma_r$ . The accuracy of our estimate  $\widehat{M}$  of  $M$  will be assessed using the following criteria:
(i) Singular subspace recovery. Let the SVD of  $\widehat{M}$  be  $\widehat{U} \widehat{\Sigma} \widehat{V}^{\top}$ . To understand how well the singular subspaces of  $M$  are recovered, we will upper bound  $\min_{O \in \mathcal{O}^{r \times r}} \|U - \widehat{U} O\|_{2 \to \infty}$  and  $\min_{O \in \mathcal{O}^{r \times r}} \|V - \widehat{V} O\|_{2 \to \infty}$  (the  $\min_{O \in \mathcal{O}^{r \times r}}$  problem corresponds to the orthogonal Procrustes problem and its solution aligns  $\widehat{U}$  and  $U$  as closely as possible, see Remark 4.1 in [14]).
(ii) Matrix estimation. To assess the accuracy of  $\widehat{M}$ , we will upper bound the row-wise error  $\| \widehat{M} - M \|_{1 \to \infty}$  or  $\| \widehat{M} - M \|_{2 \to \infty}$ , as well as the entry-wise error  $\| \widehat{M} - M \|_{\infty}$  (the spectral error  $\| \widehat{M} - M \|$  is easier to deal with and is presented in appendix only).
We introduce two classical quantities characterizing the heterogeneity and incoherence of the matrix  $M$  [11, 48]. Let  $\kappa = \sigma_1 / \sigma_r$ , and let  $\mu(U) = \sqrt{m / r} \| U \|_{2 \to \infty}$  (resp.  $\mu(V) = \sqrt{n / r} \| V \|_{2 \to \infty}$ ) denote the row-incoherence (resp. column-incoherence) parameter of  $M$ . Let  $\mu = \max \{\mu(U), \mu(V)\}$ . Next, we specify the matrices  $M$  of interest in low-rank bandits and RL, and the way the data used for their estimation is generated.
Model I: Reward matrices in low-rank bandits. For bandit problems,  $M$  corresponds to the average rewards of various arms. To estimate  $M$ , the learner has access to data sequentially generated as follows. In each round  $t = 1,\dots ,T$ , an arm  $(i_t,j_t)\in [m]\times [n]$  is randomly selected (say uniformly at random for simplicity) and the learner observes  $M_{i_t,j_t} + \xi_t$ , an unbiased sample of the corresponding entry of  $M$ .  $(\xi_{t})_{t\geq 1}$  is a sequence of zero-mean and bounded random variables. Specifically, we assume that for all  $t\geq 1$ ,  $|\xi_t|\leq c_1\| M\|_{\infty}$  a.s., for some constant  $c_{1} > 0$ .
Model II: Transition matrices in low-rank MDPs. In low-rank MDPs, we encounter Markov chains whose transition matrices have low rank  $r$  (refer to Section 5 for details). Let  $P \in \mathbb{R}^{n \times n}$  be such a transition matrix. We assume that the corresponding Markov chain is irreducible with stationary distribution  $\nu$ . The objective is to estimate  $P$  from the data consisting of samples of transitions of the chain. More precisely, from the data, we will estimate the long-term frequency matrix  $M = \mathrm{diag}(\nu)P$  ( $M_{ij}$  is the limiting proportion of transitions from state  $i$  to state  $j$  as the trajectory grows large). Observe that  $M$  is of rank  $r$ , and that  $P_{i,:} = M_{i,:} / \| M_{i,:}\|_1$ . To estimate  $M$ , the learner has access to the data  $(x_1,\ldots ,x_T) \in [n]^T$  generated according to one of the following two models.
(a) In the generative model, for any  $t \in [T]$ , if  $t$  is odd,  $x_{t}$  is selected at random according to some distribution  $\nu_{0}$ , and  $x_{t+1}$  is sampled from  $P_{x_{t};\cdot}$ .
(b) In the forward model, the learner has access to a trajectory  $(x_{1},\ldots ,x_{T})$  of length  $T$  of the Markov chain, where  $x_{1}\sim \nu_{0}$  and for any  $t\geq 1$ ,  $x_{t + 1}\sim P_{x_{t};}$ .
3 Matrix Estimation via Spectral Decomposition
In the three models (Models I, II(a) and II(b)), we first construct a matrix  $\widetilde{M}$  directly from the data, and from there, we build our estimate  $\widehat{M}$ , typically obtained via spectral decomposition, i.e., by taking the best rank- $r$  approximation of  $\widetilde{M}$ . In the remaining of this section, we let  $\widehat{U} \widehat{\Sigma} \widehat{V}^{\top}$  denote the SVD of  $\widehat{M}$ . Next, we describe in more details how  $\widehat{M}$  is constructed in the three models, and analyze the corresponding estimation error.
3.1 Reward matrices
For Model I, for  $t = 1,\dots ,T$ , we define  $\widetilde{M}_t = \big((M_{i_t,j_t} + \xi_t)\mathbb{1}_{\{(i,j) = (i_t,j_t)\}}\big)_{i,j\in [m]\times [n]}$  and  $\widetilde{M} = \frac{nm}{T}\sum_{t = 1}^{T}\widetilde{M}_t$ . Let  $\widehat{M}$  denote the best rank- $r$  approximation of  $\widetilde{M}$ .
Theorem 1. Let  $\delta >0$ . We introduce:
$$
\mathcal {B} = \sqrt {\frac {n m}{T}} \left(\sqrt {(n + m) \log \left(\frac {e (n + m) T}{\delta}\right)} + \log^ {3 / 2} \left(\frac {e (n + m) T}{\delta}\right)\right).
$$
Assume that  $T \geq c\mu^4\kappa^2 r^2 (n + m)\log^3 (e(m + n)T / \delta)$  for some universal constant  $c > 0$ . Then there exists a universal constant  $C > 0$  such that the following inequalities hold with probability at least  $1 - \delta$ :
(i)  $\max \left(\| U - \widehat{U} (\widehat{U}^{\top}U)\|_{2\to \infty},\| V - \widehat{V} (\widehat{V}^{\top}V)\|_{2\to \infty}\right)\leq C\frac{(\mu^{3}\kappa^{2}r^{3 / 2})}{\sqrt{mn(n\wedge m)}}\mathcal{B},$
(ii)  $\| \widehat{M} - M \|_{2 \to \infty} \leq C \frac{\left( \mu^3 \kappa^2 r^{3/2} \right)}{\sqrt{m \wedge n}} \| M \|_{\infty} \mathcal{B},$
(iii)  $\| \widehat{M} - M \|_{\infty} \leq C \left( \mu^{11/2} \kappa^2 r^{1/2} + \mu^3 \kappa r^{3/2} \frac{m + n}{\sqrt{mn}} \right) \frac{1}{(n \wedge m)} \| M \|_{\infty} \mathcal{B}$ .
Corollary 2. (Homogeneous reward matrix) When  $m = \Theta(n)$ ,  $\kappa = \Theta(1)$ ,  $\mu = \Theta(1)$ ,  $\|M\|_{\infty} = \Theta(1)$ ,  $r = \Theta(1)$ , we say that the reward matrix  $M$  is homogeneous. In this case, for any  $\delta > 0$ , when  $T \geq c(n + m)\log^3(e(m + n)T/\delta)$  for some universal constant  $c > 0$ , we have with probability at least  $1 - \delta$ :
$$
\max  \left(\| U - \widehat {U} (\widehat {U} ^ {\top} U) \| _ {2 \rightarrow \infty}, \| V - \widehat {V} (\widehat {V} ^ {\top} V) \| _ {2 \rightarrow \infty}\right) \lesssim \frac {1}{\sqrt {T}} \log^ {3 / 2} \left(\frac {(n + m) T}{\delta}\right),
$$
$$
\| \widehat {M} - M \| _ {2 \rightarrow \infty} \lesssim \frac {(n + m)}{\sqrt {T}} \log^ {3 / 2} \left(\frac {(n + m) T}{\delta}\right),
$$
$$
\| \widehat {M} - M \| _ {\infty} \lesssim \sqrt {\frac {(n + m)}{T}} \log^ {3 / 2} \left(\frac {(n + m) T}{\delta}\right).
$$
For a homogeneous reward matrix,  $\| U \|_{2 \to \infty} = \Theta (1 / \sqrt{m})$  and  $\| M \|_{\infty} = \Theta (1)$ , and hence, from the above corollary, we obtain estimates whose relative errors (e.g.,  $\| \widehat{M} - M \|_{\infty} / \| M \|_{\infty}$ ) scale at most as  $\sqrt{m / T}$  up to the logarithmic factor.
We may also compare the results of the above corollary to those of Theorem 4.4 presented in [14]. There, the data consists for each pair  $(i,j)$  of a noisy observation  $M_{i,j} + E_{i,j}$ . The  $E_{i,j}$ 's are independent across  $(i,j)$ . This model is simpler than ours and does not include any correlation in the data. But it roughly corresponds to the case where  $T = nm$  in our Model I. Despite having to deal with correlations, we obtain similar results as those of Theorem 4.4: for example,  $\| \widehat{M} - M \|_{\infty} \lesssim \sqrt{1 / (n + m)}$  (up to logarithmic terms) with high probability.
3.2 Transition matrices under the generative model
For Model II(a), the matrix  $\widetilde{M}$  records the empirical frequencies of the transitions: for any pair of states  $(i,j)$ ,  $\widetilde{M}_{i,j} = \frac{1}{[T/2]} \sum_{k=1}^{\lfloor T/2 \rfloor} \mathbb{1}_{\{(x_{2k-1}, x_{2k})=(i,j)\}}$ .  $\widehat{M}$  is the best rank- $r$  approximation of  $\widetilde{M}$  and the estimate  $\widehat{P}$  of the transition matrix  $P$  is obtained normalizing the rows of  $\widehat{M}$ : for all  $i \in [n]$ ,
$$
\widehat {P} _ {i,:} = \left\{ \begin{array}{l l} \left(\widehat {M} _ {i,:}\right) _ {+} / \left\| \left(\widehat {M} _ {i,:}\right) _ {+} \right\| _ {1}, & \text {i f} \left\| \left(\widehat {M} _ {i,:}\right) _ {+} \right\| _ {1} > 0, \\ \frac {1}{n} \mathbf {1} _ {n}, & \text {i f} \left\| \left(\widehat {M} _ {i,:}\right) _ {+} \right\| _ {1} = 0. \end{array} \right. \tag {1}
$$
where  $(\cdot)_{+}$  is the function applying  $\max(0, \cdot)$  component-wise and  $\mathbf{1}_n$  is the  $n$ -dimensional vector of ones. The next theorem is a simplified version and a consequence of a more general and tighter theorem presented in App. B.2. To simplify the presentation of our results, we define
$$
g (M, T, \delta) = n \log (\frac {n \sqrt {T}}{\delta}) \max \left\{\mu^ {6} \kappa^ {6} r ^ {3}, \frac {\log (\frac {n \sqrt {T}}{\delta}) \mathbb {1} _ {\{\exists \ell : T \| M _ {\ell , :} \| _ {\infty} \leq 1 \}}}{\log (1 + \frac {1}{T \| M \| _ {\infty}})} \right\}.
$$
Theorem 3. Let  $\delta >0$ . Introduce  $\mathcal{B} = \mu \kappa \sqrt{(r\|M\|_{\infty} / T)\log(n\sqrt{T} / \delta)}$ . Assume that we have  $(\nu_0)_{\min} = \min_{i\in [n]}(\nu_0)_i > 0$ . If (a)  $n\geq c\log^2 (nT^{3 / 2} / \delta)$  and (b)  $T\geq cg(M,T,\delta)$  for some universal constant  $c > 0$ , then there exists a universal constant  $C > 0$  such that the following inequalities hold with probability at least  $1 - \delta$ :
$$
\max  \left\{\| U - \widehat {U} (\widehat {U} ^ {\top} U) \| _ {2 \rightarrow \infty}, \| V - \widehat {V} (\widehat {V} ^ {\top} V) \| _ {2 \rightarrow \infty} \right\} \leq C \frac {\kappa \mu^ {2} r}{n \| M \| _ {\infty}} \mathcal {B}, \tag {i}
$$
$$
\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \hat {M} - M \| _ {2 \rightarrow \infty} \leq C \kappa B, \quad \| \hat {P} - P \| _ {1 \rightarrow \infty} \leq C \frac {\kappa \sqrt {n}}{\left(\nu_ {0}\right) _ {\min }}\right) \mathcal {B}, \right|\right|\right|\right|\right|\right|\right|\right|\right|\right|\right|\right|\right|\right|\right|\right|\right|\right|\right|\right|\right|
$$
$$
(i i i) \quad \| \widehat {M} - M \| _ {\infty} \leq C \frac {\kappa \mu^ {2} r}{\sqrt {n}} \mathcal {B},
$$
$$
\left(i v\right) \qquad \| \widehat {P} - P \| _ {\infty} \leq C \frac {\mathcal {B}}{\left(\nu_ {0}\right) _ {\min }} \left[ \sqrt {n} \kappa \frac {\| M \| _ {\infty}}{\left(\nu_ {0}\right) _ {\min }} + \left(1 + \frac {\kappa \mathcal {B}}{\sqrt {n} \| M \| _ {\infty}}\right) \frac {\kappa \mu^ {2} r}{\sqrt {n}} \right],
$$
where (iv) holds if in addition  $T \geq cn\| M\|_{\infty}(\nu_0)_{\min}^{-2}r\mu^2\kappa^4\log (n\sqrt{T} /\delta)$
Note that in theorem, the condition (a) on  $n$  has been introduced just to simplify the expression of  $\mathcal{B}$  (refer to App. B.2 for a full statement of the theorem without this condition).
Corollary 4. (Homogeneous transition matrix) When  $\kappa = \Theta(1)$ ,  $\mu = \Theta(1)$ ,  $r = \Theta(1)$ ,  $M_{\max} = \Theta(M_{\min})$ , we say that the frequency matrix  $M$  is homogeneous. If  $T \geq cn\log(nT)$  for some universal constant  $c > 0$ , then we have with probability at least  $1 - \min\{n^{-2}, T^{-1}\}$ :
$$
\max \left\{\| U - \widehat {U} (\widehat {U} ^ {\top} U) \| _ {2 \to \infty}, \| V - \widehat {V} (\widehat {V} ^ {\top} V) \| _ {2 \to \infty} \right\} \lesssim \sqrt {\frac {\log (n T)}{T}},
$$
$$
\| \widehat {M} - M \| _ {2 \to \infty} \lesssim \frac {1}{n} \sqrt {\frac {\log (n T)}{T}}, \| \widehat {M} - M \| _ {\infty} \lesssim \frac {1}{n} \sqrt {\frac {\log (n T)}{n T}},
$$
$$
\| \hat {P} - P \| _ {1 \rightarrow \infty} \lesssim \sqrt {\frac {n \log (n T)}{T}}, \| \hat {P} - P \| _ {\infty} \lesssim \sqrt {\frac {\log (n T)}{n T}}.
$$
For a homogeneous frequency matrix,  $\| U \|_{2 \to \infty} = \Theta(1 / \sqrt{n})$ ,  $\| M \|_{2 \to \infty} = \Theta(1 / n \sqrt{n})$ ,  $\| M \|_{\infty} = \Theta(1 / n^2)$ ,  $\| P \|_{1 \to \infty} = 1$ ,  $\| P \|_{\infty} = \Theta(1 / n)$ . Thus for all these metrics, our estimates achieve a relative error scaling at most as  $\sqrt{n / T}$  up to the logarithmic factor.
3.3 Transition matrices under the forward model
For Model II(b), we first split the data into  $\tau$  subsets of transitions: for  $k = 1, \ldots, \tau$ , the  $k$ -th subset is  $((x_k, x_{k+1}), (x_{k+\tau}, x_{k+1+\tau}), \ldots, (x_{k+(T_\tau-1)\tau}, x_{k+1+(T_\tau-1)\tau}))$  where  $T_\tau = \lfloor T / \tau \rfloor$ . By separating two transitions in the same subset, we break the inherent correlations in the data if  $\tau$  is large enough. Now we let  $\widetilde{M}^{(k)}$  be the matrix recording the empirical frequencies of the transitions in the  $k$ -th subset:  $\widetilde{M}_{i,j}^{(k)} = \frac{1}{T_\tau} \sum_{l=0}^{T_\tau-1} \mathbb{1}_{\{(x_{k+l\tau}, x_{k+1+l\tau})=(i,j)\}}$  for any pair of states  $(i,j)$ . Let  $\widehat{M}^{(k)}$  be the best  $r$ -rank approximation of  $\widetilde{M}^{(k)}$ . As in (1), we define the corresponding  $\widehat{P}^{(k)}$ . Finally we may aggregate these estimates  $\widehat{M} = \frac{1}{\tau} \sum_{k=1}^{\tau} \widehat{M}^{(k)}$  and  $\widehat{P} = \frac{1}{\tau} \sum_{k=1}^{\tau} \widehat{P}^{(k)}$ . We present below the performance analysis for the estimates coming from a single subset; the analysis of the aggregate estimates easily follows.
For any  $\varepsilon > 0$ , we define the  $\varepsilon$ -mixing time of the Markov chain with transition matrix  $P$  as  $\tau(\varepsilon) = \min \{t \geq 1 : \max_{1 \leq i \leq n} \frac{1}{2} \| P_{i,:}^t - \nu^\top \|_1 \leq \varepsilon \}$ , and its mixing time as  $\tau^\star = \tau(1/4)$ . The next theorem is a simplified version and a consequence of a more general and tighter theorem presented in App. B.3. To simplify the presentation, we define:
$$
h (M, T, \delta) = n \tau^ {\star} \log \left(\frac {n \sqrt {T}}{\delta}\right) \log \left(T \nu_ {\min } ^ {- 1}\right) \max  \left\{\mu^ {6} \kappa^ {6} r ^ {3}, \frac {\log^ {2} \left(\frac {n \sqrt {T _ {\tau}}}{\delta}\right) \mathbb {1} _ {\{\exists \ell : T _ {\tau} \| M _ {\ell , :} \| _ {\infty} \leq 1 \}}}{\log^ {2} \left(1 + \frac {1}{T _ {\tau} \| M \| _ {\infty}}\right)} \right\}.
$$
Theorem 5. Let  $\delta >0$ . Assume that  $\nu_{\mathrm{min}} = \min_{i\in [n]}\nu_{i} > 0$  and that  $\tau /(\tau^{\star}\log (T\nu_{\mathrm{min}}^{-1}))\in [c_1,c_2]$  for some universal constants  $c_{2} > c_{1}\geq 2$ . Introduce:
$$
\mathcal {B} = \mu \kappa \sqrt {\frac {r \tau^ {\star} \| M \| _ {\infty}}{T} \log \left(\frac {n \sqrt {T _ {\tau}}}{\delta}\right) \log \left(\frac {T}{\nu_ {\mathrm {m i n}}}\right)}.
$$
If  $(a) n \geq c\tau^{\star} \log^{3/2}(n T^{3/2} / \delta) \log^{1/2}(T\nu_{\min}^{-1})$  and  $(b) T \geq c h(M, T, \delta)$  for some universal constant  $c > 0$ , then there exists a universal constant  $C > 0$  such that the following inequalities hold with probability at least  $1 - \delta$ :
$$
(i) \quad \max  \left\{\| U - \widehat {U} (\widehat {U} ^ {\top} U) \| _ {2 \rightarrow \infty}, \| V - \widehat {V} (\widehat {V} ^ {\top} V) \| _ {2 \rightarrow \infty} \right\} \leq C \frac {\kappa \mu^ {2} r}{n \| M \| _ {\infty}} \mathcal {B},
$$
$$
\left(i i\right) \quad \| \widehat {M} - M \| _ {2 \rightarrow \infty} \leq C \kappa \mathcal {B}, \quad \| \widehat {P} - P \| _ {1 \rightarrow \infty} \leq C \frac {\kappa \sqrt {n}}{\nu_ {\min }} \mathcal {B},
$$
$$
\begin{array}{l l} (i i i) & \| \widehat {M} - M \| _ {\infty} \leq C \frac {\kappa \mu^ {2} r}{\sqrt {n}} \mathcal {B}, \end{array}
$$
$$
\left(i v\right) \qquad \| \widehat {P} - P \| _ {\infty} \leq C \frac {\mathcal {B}}{\nu_ {\mathrm {m i n}}} \left[ \sqrt {n} \kappa \frac {\| M \| _ {\infty}}{\nu_ {\mathrm {m i n}}} + \left(1 + \frac {\kappa \mathcal {B}}{\sqrt {n} \| M \| _ {\infty}}\right) \frac {\kappa \mu^ {2} r}{\sqrt {n}} \right],
$$
where (iv) holds if in addition  $T \geq cn\| M\|_{\infty}\nu_{\min}^{-2}\tau^{\star}r\mu^{2}\kappa^{4}\log (n\sqrt{T} /\delta)\log (T\nu_{\min}^{-1}).$
Note that our guarantees hold when  $\tau$  roughly scales as  $\tau^{\star}\log (T\nu_{\mathrm{min}}^{-1})$ . Hence to select  $\tau$ , one would need an idea of the latter quantity. It can be estimated typically using  $\tau^{\star}\nu_{\mathrm{min}}^{-1}$  samples [61] (which
is small when compared to the constraint  $T \geq ch(M, T, \delta)$  as soon as  $\nu_{\min} = \Omega(1/n)$ . Further observe that in the theorem, the condition (a) can be removed (refer to App. B.3 for a full statement of the theorem without this condition).
Corollary 6. (Homogeneous transition matrices) Assume that  $M$  is homogeneous (as defined in Corollary 4). Let  $\tau = \log(Tn)$ . If  $T \geq cn\log^2(nT)$  for some universal constant  $c > 0$ , then we have with probability at least  $1 - \min\{n^{-2}, T^{-1}\}$ :
$$
\max \left\{\| U - \widehat {U} (\widehat {U} ^ {\top} U) \| _ {2 \to \infty}, \| V - \widehat {V} (\widehat {V} ^ {\top} V) \| _ {2 \to \infty} \right\} \lesssim \frac {1}{\sqrt {T}} \log (n T),
$$
$$
\| \widehat {M} - M \| _ {2 \rightarrow \infty} \lesssim \frac {1}{n \sqrt {T}} \log (n T), \| \widehat {M} - M \| _ {\infty} \lesssim \frac {1}{n \sqrt {n T}} \log (n T),
$$