SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Then our algorithm will further interact with the environment in successive two steps after collecting  $s_k \sim d_{P^{\star}}^{\tilde{\pi}_k}$  (Line 7 - Line 11) like previous works studying low-rank MDPs [Agarwal et al., 2020, Uehara et al., 2022, Zhang et al., 2022]. One of the main differences between previous algorithms and ours lies in how to deal with exploration and exploitation when interacting with the environment. In the case of stochastic low-rank MDPs, the key in the analysis is to ensure the (near) optimism of the optimal policy  $\pi^{\star}$  in the learned model, which essentially requires to bound the performance gap  $\widehat{V}_k^{\pi^{\star}} - V_k^{\pi^{\star}}$  by relating it with the model error regarding  $\pi^{\star}$ , i.e.,  $\mathbb{E}_{(s,a) \sim d_{P^{\star}}^{\pi^{\star}}}[\| \hat{P}_k(\cdot | s,a) - P^{\star}(\cdot | s,a) \|_1]$ , where  $\hat{P}_k$  is the MLE solution defined in Eq. (1). However, this model error is not directly controllable, as the algorithm does not know the optimal policy  $\pi^{\star}$  and can not collect data following  $\pi^{\star}$  to bound the model error. Fortunately, by empirical process theory [Geer, 2000, Zhang, 2006], the model error  $\mathbb{E}_{(s,a) \sim \rho_k}[\| \hat{P}_k(\cdot | s,a) - P^{\star}(\cdot | s,a) \|_1]$  regarding the executed policies  $\{\tilde{\pi}_i\}_{i=1}^k$  is bounded, where  $\rho_k(s,a) = \frac{1}{k} \sum_{i=1}^{k} d_{P^{\star}}^{\tilde{\pi}_i}(s,a)$ . Therefore, previous works [Agarwal et al., 2020, Uehara et al., 2022, Zhang et al., 2022] ensure the optimism of the optimal policy  $\pi^{\star}$  by applying importance weighting to change the measure from  $d_{P^{\star}}^{\pi^{\star}}$  to  $\rho_k$  when bounding the model error. The complication is that to control the ratio  $\pi^{\star}(a | s) / \tilde{\pi}_i(a | s)$  for any  $(s,a) \in S \times A$  and  $i \in [K]$  in importance weighting, the algorithms in previous works take actions from the uniform distribution  $U(A)$  over action space  $A$ , which intuitively can be seen as conducting exploration over transitions required by representation learning. Consequently, though these algorithms enjoy excellent sample complexities, they intrinsically do not have regret guarantees due to the uniform exploration over action space, even in the stochastic setting. Moreover, to learn adversarial low-rank MDPs, it is required to take actions adaptively according to the observed loss functions in previous episodes instead of uniformly taking actions. To address this "conflict" so as to learn adversarial low-rank MDPs, we propose to use a mixed roll-out policy to interleave (a) the exploration over transitions required by representation learning; and (b) the exploration and exploitation over the adversarial loss functions by policy optimization, which we call **doubled exploration** and **exploitation** and is pivotal to achieving our regret bound as we will shortly see. Formally, our algorithm will conduct the exploration over the transitions with probability  $\xi$  and execute policy  $\tilde{\pi}_k$  optimized by OMD with probability  $1 - \xi$  respectively, as shown in Line 7 - Line 11.
After interacting with the environment, the newly collected data in these two steps will be used to update the datasets (Line 14), and the empirical transition  $\widehat{P}_k$  will be updated by performing MLE over the updated datasets by solving (Line 16)
$$
\left(\widehat {\mu} _ {k}, \widehat {\phi} _ {k}\right) = \underset {(\mu , \phi) \in \mathcal {M}} {\arg \max } \mathbb {E} _ {\mathcal {D} _ {k} \cup \mathcal {D} _ {k} ^ {\prime}} \left[ \ln \mu^ {\top} \left(s ^ {\prime}\right) \phi (s, a) \right], \tag {1}
$$
where we denote  $\mathbb{E}_{\mathcal{D}}[f(s,a,s^{\prime})] = \frac{1}{|\mathcal{D}|}\sum_{(s,a,s^{\prime})\in \mathcal{D}}f(s,a,s^{\prime}).$
Algorithm 1 Policy Optimization for Low-rank MDPs (POLO)
1: Input: Mixing coefficient  $\xi$ , epoch length  $L$ , regularization coefficients  $\{\lambda_k\}_{k=1}^K$ , bonus coefficients  $\{\alpha_k\}_{k=1}^K$ , model class  $\mathcal{M}$ , number of episodes  $K$ , learning rate  $\eta$ .  
2: Initialization: Set  $\mathcal{D}_0 = \emptyset$ ,  $\mathcal{D}_0' = \emptyset$ .  
3: for  $i = 1,2,\ldots,\lceil K/L \rceil$  do  
4: Set  $k_i = (i-1)L+1$  and  $\tilde{\pi}_{k_i}(\cdot|s)$  to be uniform for any  $s \in S$ .  
5: for  $k = k_i, k_i+1,\ldots,k_i+L-1$  do  
6: Sample  $sk$  from  $d_{P^*}'$ .  
7: Sample  $ck \sim \mathrm{Ber}(1-\xi)$ .  
8: if  $ck = 1$  then  
9: Sample  $ak \sim \tilde{\pi}_k(\cdot|sk), s'_k \sim P^*(\cdot|sk, a_k), a'_k \sim \tilde{\pi}_k(\cdot|s'_k), s'_k \sim P^*(\cdot|s'_k, a'_k)$ .  
10: else  
11: Sample  $ak \sim U(\mathcal{A}), s'_k \sim P^*(\cdot|sk, a_k), a'_k \sim U(\mathcal{A}), s'_k \sim P^*(\cdot|s'_k, a'_k)$ .  
12: end if  
13: Observe the loss function  $\ell_k$ .  
14: Update datasets  $\mathcal{D}_k = \mathcal{D}_{k-1} \cup \{(s_k, a_k, s'_k)\}$ ,  $\mathcal{D}_k' = \mathcal{D}_{k-1}' \cup \{(s'_k, a'_k, s'_k)\}$ .  
15: if  $k = k_i$  then  
16: Set the empirical transition  $\widehat{P}_k(s'|s,a) = \widehat{\mu}_k(s')^\top \widehat{\phi}_k(s,a), \forall (s,a,s') \in S \times A \times S$ , via solving Eq. (1).  
17: Update the empirical covariance matrix  $\widehat{\Sigma}_k = \sum_{(s,a) \in \mathcal{D}_k} \widehat{\phi}_k(s,a) \widehat{\phi}_k(s,a)^\top + \lambda_k I$ .  
18: Set the bonus function  $\widehat{b}_k(s,a) := \min(\alpha_k \| \widehat{\phi}_k(s,a) \|_{\widehat{\Sigma}_k^{-1}}, 2) / (1 - \gamma), \forall (s,a) \in S \times A$ .  
19: else  
20: Set the empirical transition  $\widehat{P}_k = \widehat{P}_{ki}$  and bonus function  $\widehat{b}_k = \widehat{b}_{ki}$ .  
21: end if  
22: Compute  $\widehat{Q}_k^{\tilde{\pi}_k}(\cdot,\cdot) = \text{Policy-Evaluation}(\widehat{P}_k, \ell_k - \widehat{b}_k, \tilde{\pi}_k)$ .  
23: Update policy  $\tilde{\pi}_{k+1}(\cdot|\cdot) \propto \tilde{\pi}_k(\cdot|\cdot) \exp(-\eta \widehat{Q}_k^{\tilde{\pi}_k}(\cdot,\cdot))$ .  
24: end for  
25: end for
3.2 Policy Optimization in Fixed Learned Models
It remains to compute the policy  $\tilde{\pi}_{k+1}$  to be used in the next episode. To this end, we resort to the canonical OMD framework to perform policy optimization like previous methods [Shani et al., 2020a, Cai et al., 2020, He et al., 2022]. However, previous OMD-based policy optimization methods for tabular and linear (mixture) MDPs [Shani et al., 2020a, Cai et al., 2020, He et al., 2022] critically depend on the point-wise optimism for each state-action pair, i.e.,  $\widehat{Q}_k^{\tilde{\pi}_k}(s,a)\leq \ell_k(s,a) + \gamma [P^\star \widehat{V}_k^{\tilde{\pi}_k}](s,a)$ , to enable the decomposition (cf., Lemma 1 by Shani et al. [2020a])
$$
\begin{array}{l} \widehat {V} _ {k} ^ {\tilde {\pi} _ {k}} (s _ {0}) - V _ {k} ^ {\pi^ {\star}} (s _ {0}) = \mathbb {E} \left[ \sum_ {\tau = 0} ^ {\infty} \gamma^ {\tau} \left\langle \tilde {\pi} _ {k} (\cdot | s _ {\tau}) - \pi^ {\star} (\cdot | s _ {\tau}), \widehat {Q} _ {k} ^ {\tilde {\pi} _ {k}} (s _ {\tau}, \cdot) \right\rangle \Bigg | \pi^ {\star}, P ^ {\star}, s _ {0} \right] \\ + \mathbb {E} \left[ \sum_ {\tau = 0} ^ {\infty} \gamma^ {\tau} \left(\widehat {Q} _ {k} ^ {\tilde {\pi} _ {k}} \left(s _ {\tau}, a _ {\tau}\right) - \ell_ {k} \left(s _ {\tau}, a _ {\tau}\right) - \gamma \left[ P ^ {\star} \widehat {V} _ {k} ^ {\tilde {\pi} _ {k}} \right] \left(s _ {\tau}, a _ {\tau}\right)\right) \mid \pi^ {\star}, P ^ {\star}, s _ {0} \right], \\ \end{array}
$$
where  $\widehat{V}_k^{\widetilde{\pi}_k}$  and  $\widehat{Q}_k^{\widetilde{\pi}_k}$  are the state value and state-action value functions of  $\widetilde{\pi}_k$  on  $(\widehat{P}_k, \ell_k - \widehat{b}_k)$  with  $\widehat{b}_k$  as some bonus function and the expectation is taken over the randomness of sampling  $a_{\tau} \sim \pi^{\star}(\cdot \mid s_{\tau})$  and  $s_{\tau + 1} \sim P^{\star}(\cdot \mid s_{\tau}, a_{\tau})$ . The summation of the first term in the above display is contributed by competing with the optimal policy  $\pi^{\star}$  in the true model  $P^{\star}$  and can be bounded by common OMD analysis, which thus can be regarded as conducting policy optimization in the true model. The point-wise optimism guarantees that the second term is less than or equal to 0.
Nevertheless, in low-rank MDPs, due to the unknown representation, it is generally hard to obtain the above point-wise optimism, which leaves the second optimism term unbounded. To cope with this issue, we instead consider the following decomposition:
$$
\widehat {V} _ {k} ^ {\tilde {\pi} _ {k}} \left(s _ {0}\right) - V _ {k} ^ {\pi^ {\star}} \left(s _ {0}\right)
$$
$$
\begin{array}{l} = \widehat {V} _ {k} ^ {\bar {\pi} _ {k}} (s _ {0}) - \widehat {V} _ {k} ^ {\pi^ {\star}} (s _ {0}) + \widehat {V} _ {k} ^ {\pi^ {\star}} (s _ {0}) - V _ {k} ^ {\pi^ {\star}} (s _ {0}) \\ = \mathbb {E} \left[ \sum_ {\tau = 0} ^ {\infty} \gamma^ {\tau} \left\langle \tilde {\pi} _ {k} (\cdot \mid s _ {k, \tau}) - \pi^ {\star} (\cdot \mid s _ {k, \tau}), \widehat {Q} _ {k} ^ {\tilde {\pi} _ {k}} (s _ {k, \tau}, \cdot) \right\rangle \mid \pi^ {\star}, \widehat {P} _ {k}, s _ {0} \right] + \widehat {V} _ {k} ^ {\pi^ {\star}} (s _ {0}) - V _ {k} ^ {\pi^ {\star}} (s _ {0}), \tag {2} \\ \end{array}
$$
where the first term is contributed by competing against the optimal policy  $\pi^{\star}$  in the learned model  $\widehat{P}_k$  and can be seen as conducting policy optimization in learned models. This decomposition will be amenable as long as we can achieve a near optimism at the initial state  $s_0$ , i.e.,  $\widehat{V}_k^{\pi^{\star}}(s_0) - V_k^{\pi^{\star}}(s_0) \lesssim 0$ , which turns out to be feasible for low-rank MDPs [Uehara et al., 2022]. However, there remains one more caveat. The first term in Eq. (2) is now no longer directly bounded by OMD analysis, due to the local update nature of OMD-based policy optimization at each state and the state occupancy distribution  $d_{\widehat{P}_k}^{\pi^{\star}}$  now varies across different episodes. To address this issue, Algorithm 1 adopts an epoch-based transition update, in which one epoch has  $L$  episodes and the model is only updated at the first episode in one epoch (Line 15 - Line 20). Concretely, Algorithm 1 sets  $\widehat{P}_k = \widehat{P}_{k_i}$  and  $\widehat{b}_k = \widehat{b}_{k_i}$ , where  $k_i$  is the first episode of the epoch to which the episode  $k$  belongs. In this manner, the learned model is fixed in one epoch, and thus the regret of dealing with the adversarial loss functions by competing against the optimal policy  $\pi^{\star}$  can be bounded in one epoch. Subsequently, at the end of episode  $k$ , our algorithm first computes the optimistic value estimate  $\widehat{Q}_{k}^{\widetilde{\pi}_{k}}$  for current policy  $\widetilde{\pi}_k$  under  $\widehat{P}_k$  together with the bonus-enhanced loss functions  $\ell_k - \widehat{b}_k$  by policy evaluation (Line 22). Note that this boils down to planning in the setting of linear MDPs for given features in the learned model and this can be done computationally efficiently [Jin et al., 2020b]. Then the policy is updated by solving
$$
\tilde {\pi} _ {k + 1} (\cdot \mid s) \in \underset {\pi (\cdot \mid s) \in \Delta (\mathcal {A})} {\arg \min } \eta \left\langle \pi (\cdot \mid s), \widehat {Q} _ {k} ^ {\tilde {\pi} _ {k}} (s, \cdot) \right\rangle + D _ {F} (\pi (\cdot \mid s), \tilde {\pi} _ {k} (\cdot \mid s)), \tag {3}
$$
where  $\eta > 0$  is the learning rate to be tuned later and  $D_F(x, y) = F(x) - F(y) - \langle x - y, \nabla F(y) \rangle$  is the Bregman divergence induced by the regularizer  $F$ . With  $F(\pi(\cdot | s)) = \sum_{a \in \mathcal{A}} \pi(a | s) \ln \pi(a | s)$  as the negative entropy, the closed-form solution to the above display is shown in Line 23, which can be regarded as a kind of soft policy improvement.
4.1 Regret Upper Bound
The regret upper bound of our POLO algorithm for learning adversarial low-rank MDPs is guaranteed by the following theorem.
Theorem 4.1. Suppose  $K > d^{6}A^{3} / (1 - \gamma)^{6}$ . For any adversarial low-rank MDP satisfying Definition 2.1, by setting the epoch length  $L = K^{1/2}A^{-1/2}d^{-1}\xi(1 - \gamma)$ , learning rate  $\eta = (1 - \gamma)\sqrt{\ln A/(2L)}$ , bonus coefficient  $\alpha_{k} = O(\sqrt{\gamma(A/\xi + d^{2})\ln(Mk/\delta)})$ , regularization coefficient  $\lambda_{k} = O(d\ln(Mk/\delta))$ , mixing coefficient  $\xi = K^{-1/6}A^{1/2}d/(1 - \gamma)$ , and  $\delta = 1/K$ , then the regret of Algorithm 1 is upper bounded by
$$
\mathcal {R} _ {K} = O \left(\frac {K ^ {\frac {5}{6}} A ^ {\frac {1}{2}} d \ln \left(1 + A M K ^ {2}\right)}{(1 - \gamma) ^ {2}}\right).
$$
Remark 4.1. Ignoring the dependence on all logarithmic factors but  $M$ , the regret upper bound can be simplified as  $\tilde{O}(K^{5/6}A^{1/2}d\ln(1 + M) / (1 - \gamma)^2)$ . Comparing with the regret lower bound  $\Omega(\frac{\gamma^2}{1 - \gamma}\sqrt{dAK})$  in Theorem 4.2, the regret upper bound in Theorem 4.1 matches in  $A$  up to a logarithmic factor but looses in factors of  $K$  and  $d$ . Also, note that when  $K$  is large enough such that  $\xi$  and  $L$  can be chosen as  $\xi = K^{-1/6}A^{1/3}d^{2/3} / (1 - \gamma)$  and  $L = K^{1/2}A^{-1}d^{-2}\xi(1 - \gamma) = K^{1/3}A^{-2/3}d^{-4/3} \geq 1$ , meaning that  $K \geq d^4A^2$ , the regret upper bound can be further optimized to  $\tilde{O}(K^{5/6}A^{1/3}d^{2/3}\ln(1 + M) / (1 - \gamma)^2)$ . Note that this does not conflict with the regret lower bound in Section 4.3 since the magnitude of this upper bound is still larger than that of the regret lower bound as long as  $K \geq A^{1/2}d^{-1/2}\gamma^6(1 - \gamma)^3$ .
4 Throughout this paper, we suppose for simplicity that the number of episodes  $K$  is divisible by the epoch length  $L$  considered.
4.2 Proof of Regret Upper Bound
We now present the proof of Theorem 4.1. To begin with, recall that in each episode  $k$ , after state  $s_k$  is sampled from  $d_{P^*}^{\tilde{\pi}_k}$ , the actual roll-out policy will be  $\pi_k(\cdot | s) = \xi \cdot U(\mathcal{A}) + (1 - \xi) \cdot \tilde{\pi}_k(\cdot | s)$ . Therefore, it holds that
$$
\begin{array}{l} \mathcal {R} _ {K} = \mathbb {E} \left[ \sum_ {k = 1} ^ {K} \left(V _ {k} ^ {\pi_ {k}} - V _ {k} ^ {\pi^ {*}}\right) \right] \\ = \mathbb {E} \left[ \sum_ {k = 1} ^ {K} \mathbb {I} \left\{c _ {k} = 1 \right\} \left(V _ {k} ^ {\pi_ {k}} - V _ {k} ^ {\pi^ {*}}\right) + \mathbb {I} \left\{c _ {k} = 0 \right\} \left(V _ {k} ^ {\pi_ {k}} - V _ {k} ^ {\pi^ {*}}\right) \right] \\ \leq \mathbb {E} \left[ \sum_ {k = 1} ^ {K} \left(V _ {k} ^ {\tilde {\pi} _ {k}} - V _ {k} ^ {\pi^ {*}}\right) \right] + \frac {\xi K}{(1 - \gamma)}, \tag {4} \\ \end{array}
$$
where the inequality is due to that  $\sum_{\tau = 0}^{\infty}\gamma^{\tau}\ell_{k}(s_{\tau},a_{\tau})\in [0,1 / (1 - \gamma)]$  holds for any episode  $k$  and any trajectory  $\{(s_{\tau},a_{\tau})\}_{\tau = 0}^{\infty}$ . We now turn to bound the first term in Eq. (4) by decomposing it into the following three terms
$$
\mathbb {E} \left[ \sum_ {k = 1} ^ {K} \left(V _ {k} ^ {\tilde {\pi} _ {k}} - V _ {k} ^ {\pi^ {*}}\right) \right] = \mathbb {E} \left[ \underbrace {\sum_ {k = 1} ^ {K} \left(V _ {k} ^ {\tilde {\pi} _ {k}} - \widehat {V} _ {k} ^ {\tilde {\pi} _ {k}}\right)} _ {\text {E S T I M A T I O N B I A S T E R M}} + \underbrace {\sum_ {k = 1} ^ {K} \left(\widehat {V} _ {k} ^ {\tilde {\pi} _ {k}} - \widehat {V} _ {k} ^ {\pi^ {*}}\right)} _ {\text {O M D R E G R E T T E R M}} + \underbrace {\sum_ {k = 1} ^ {K} \left(\widehat {V} _ {k} ^ {\pi^ {*}} - V _ {k} ^ {\pi^ {*}}\right)} _ {\text {O P T I M I S M T E R M}} \right]. \tag {5}
$$
Bounding OMD REGRET TERM The OMD REGRET TERM is contributed by competing against  $\pi^{\star}$  using  $\tilde{\pi}_k$  with  $\widehat{Q}_k^{\tilde{\pi}_k}$  as loss function in the learned model  $\widehat{P}_{k_i}$ . This term is thus bounded by standard OMD analysis, detailed in the following lemma.
Lemma 4.1. By setting  $\eta = (1 - \gamma)\sqrt{\ln A / (2L)}$ , the OMD REGRET TERM is bounded as  $\mathbb{E}\left[\sum_{k = 1}^{K}\left(\widehat{V}_k^{\widetilde{\pi}_k} - \widehat{V}_k^{\pi^\star}\right)\right] \leq \frac{K\sqrt{2\ln A}}{\sqrt{L}(1 - \gamma)^2}$ .
Bounding OPTIMISM TERM The OPTIMISM TERM is controlled by choosing appropriate bonus coefficient  $\alpha_{k}$ . Note that different from tabular and linear cases, the bonus functions and coefficients here are not devised to control the optimism for each state-action pair. Instead, they are devised to provide a (near) optimism only at the initial state  $s_0$ .
Lemma 4.2. By setting  $\alpha_{k} = O(\sqrt{\gamma(A / \xi + d^{2})\ln(Mk / \delta)})$  and  $\lambda_{k} = O(d\ln (Mk / \delta))$ , with probability  $1 - \delta$ , the OPTIMISM TERM is bounded as  $\sum_{k = 1}^{K}\left(\widehat{V}_k^{\pi^*} - V_k^{\pi^*}\right) \leq (L + \sqrt{K})\sqrt{\frac{A\ln(MN / \delta)}{\xi(1 - \gamma)^3}}$ .
Bounding ESTIMATION BIAS TERM It remains to bound the ESTIMATION BIAS TERM, which comes from the difference between the values of running the same policy  $\tilde{\pi}_k$  in the true model (i.e.,  $P^{\star}$  and  $\ell_k$ ) and the learned empirical model (i.e.,  $\widehat{P}_k$  and  $\ell_k - \widehat{b}_k$ ), respectively. This term can be translated into the error between the true model and the learned model using the common simulation lemma, which is thus bounded by the summation of bonus functions. Note that since the empirical features used to construct our bonus functions vary in each episode, we first relate the bonus functions with the fixed true feature  $\phi^{\star}$  using the one-step trick [Uehara et al., 2022, Zhang et al., 2022], and finally bound this term with the leverage of the canonical elliptical potential lemma. The result is shown in the following lemma.
Lemma 4.3. By setting  $\alpha_{k} = O(\sqrt{\gamma(A / \xi + d^{2})\ln(Mk / \delta)})$  and  $\lambda_{k} = O(d\ln (Mk / \delta))$ , with probability  $1 - \delta$ , the ESTIMATION BIAS TERM is bounded as  $\sum_{k = 1}^{K}\left(V_k^{\tilde{\pi}_k} - \widehat{V}_k^{\tilde{\pi}_k}\right) \leq O\left(\frac{d^2A\sqrt{KL}}{\xi(1 - \gamma)^3}\sqrt{\ln(1 + K)\ln(MK / \delta)}\right)$ .
We refer the readers to Appendix A for the proof of the above lemmas. The proof of Theorem 4.1 is now concluded by first combining Eq. (4), Eq. (5), Lemma 4.1, 4.2, and 4.3 and then choosing  $L = K^{1/2}A^{-1/2}d^{-1}\xi(1 - \gamma)$ ,  $\xi = K^{-1/6}A^{1/2}d/(1 - \gamma)$ , and  $\delta = 1 / K$ .
Intuitively, the epoch length  $L$  illustrates a trade-off between dealing with the adversarial losses and the representation learning over the unknown transitions. When  $L$  is large, there will be fewer restarts in the running of OMD and thus the learner will suffer less regret contributed by dealing with the adversarial losses as shown by Lemma 4.1. In contrast, a smaller  $L$  enables more frequent model updates, which leads to more accurate model estimation and less regret contributed by the representation learning as shown by Lemma 4.2 and 4.3.
4.3 Regret Lower Bound
This section presents the regret lower bound for learning adversarial low-rank MDPs with fixed loss functions in Theorem 4.2, which thus also serves as a regret lower bound for learning adversarial low-rank MDPs with full-information feedback.
Theorem 4.2. Suppose  $d \geq 8$ ,  $S \geq d + 1$ ,  $A \geq d - 3$ , and  $K \geq 2(d - 4)A$ . Then for any algorithm Alg, there exists an episodic infinite-horizon low-rank MDP  $\mathcal{M}_{\mathrm{Alg}}$  with fixed loss function such that the regret for this MDP is lower bounded by  $\Omega \left( \frac{\gamma^2}{1 - \gamma} \sqrt{dAK} \right)$ .
Proof Sketch. At a high level, we construct  $dA$  hard-to-learn low-rank MDP instances, which are difficult to distinguish in KL divergence but have very different optimal policies. In particular, all the constructed low-rank MDP instances have three levels of states, in which the only state in the first level is a fixed initial state and the states in the third level are absorbing states. Moreover, only one unique absorbing state in the third level has the lowest loss, which is termed as the "good state". In the constructed low-rank MDP instance  $\mathcal{M}_{(i^{\star},a^{\star})}$ , the learner can only take specific action to transfer to state  $s_{2,i^{\star}}$  in the second level and then take the other specific action to transfer to the unique good state. Due to the unknown representations of state-action pairs, the learner needs to distinguish all these  $dA$  low-rank MDP instances, which is essentially equivalent to dealing with a bandit problem with  $dA$  "arms". The detailed proof of Theorem 4.2 is postponed to Appendix B.
Remark 4.2. Theorem 4.2, to the best of our knowledge, provides the first regret lower bound for learning low-rank MDPs with fixed loss functions. We note that this regret lower bound can hold when  $d \ll S$  and  $d \ll A$ , which thus means that this lower bound is non-trivial. Besides, the regret upper bound in our Theorem 4.1 matches the regret lower bound in  $A$  up to a logarithmic factor but looses a factor of  $\widetilde{O}(K^{1/3}d^{1/2} / ((1 - \gamma)\gamma^2))$ . Importantly, compared with the regret upper bound  $\widetilde{O}(d\sqrt{K / (1 - \gamma)^3})$  of linear MDPs [He et al., 2023] (the finite horizon  $H$  is substituted by the effective horizon  $\Theta(1 / (1 - \gamma))$  in our infinite-horizon setting for a fair comparison), the dependence on  $A$  in the regret lower bound of low-rank MDP shows a clear separation between low-rank MDPs and linear MDPs, which demonstrates that low-rank MDPs are statistically more difficult to learn than linear MDPs in the regret minimization setting.
In this work, we study learning adversarial low-rank MDPs with unknown transition and full-information feedback. We prove that our proposed algorithm POLO achieves the  $\widetilde{O}(K^{5/6}A^{1/2}d\ln(1 + M)/(1 - \gamma)^2)$  regret, which is the first sublinear regret guarantee for this challenging problem. The design of our proposed algorithm features (a) a doubled exploration and exploitation scheme to simultaneously learn the transitions and adversarial loss functions; and (b) policy optimization in the fixed learned models with epoch-based model update to enable a new analysis scheme that only requires the near optimism at the initial state instead of the point-wise optimism. Also, we prove an  $\Omega(\frac{\gamma^2}{1 - \gamma}\sqrt{dAK})$  regret lower bound for this problem, serving as the first regret lower bound for learning low-rank MDPs in the regret minimization setting. Besides, there also remain several interesting future directions to be explored. One natural question is whether it is possible to further optimize the dependence of our regret guarantee on the number of episodes  $K$ . The other question is how to also learn adversarial low-rank MDPs with only the bandit feedback available. This is also challenging since the current occupancy measure-based methods and policy optimization-based methods tackling adversarial MDPs with bandit feedback both depend on the point-wise optimism provided by the true feature mapping, which seems not feasible in low-rank MDPs. We hope our results may shed light on better understandings of RL with both nonlinear function approximation and adversarial losses and we leave the above extensions as our future works.