SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Canzhe Zhao
canzhezhao@sjtu.edu.cn
Ruofeng Yang
wanshuiyin@sjtu.edu.cn
Baoxiang Wang
The Chinese University of Hong Kong, Shenzhen bxiangwang@cuhk.edu.cn
Xuezhou Zhang
xuezhouz@bu.edu
Shuai Li*
shuaili8@sjtu.edu.cn
In this work, we study the low-rank MDPs with adversariably changed losses in the full-information feedback setting. In particular, the unknown transition probability kernel admits a low-rank matrix decomposition [Uehara et al., 2022], and the loss functions may change adversariably but are revealed to the learner at the end of each episode. We propose a policy optimization-based algorithm POLO, and we prove that it attains the  $\widetilde{O}(K^{5/6}A^{1/2}d\ln(1 + M) / (1 - \gamma)^2)$  regret guarantee, where  $d$  is the rank of the transition kernel (and hence the dimension of the unknown representations),  $A$  is the cardinality of the action space,  $M$  is the cardinality of the model class that contains all the plausible representations, and  $\gamma$  is the discounted factor. Notably, our algorithm is oracle-efficient and has a regret guarantee with no dependence on the size of potentially arbitrarily large state space. Furthermore, we also prove an  $\Omega\left(\frac{\gamma^2}{1 - \gamma}\sqrt{dAK}\right)$  regret lower bound for this problem, showing that low-rank MDPs are statistically more difficult to learn than linear MDPs in the regret minimization setting. To the best of our knowledge, we present the first algorithm that interleaves representation learning, exploration, and exploitation to achieve the sublinear regret guarantee for RL with nonlinear function approximation and adversarial losses.
In reinforcement learning (RL), the goal is to learn a (near) optimal policy through the interactions between the learner and the environment, which is typically modeled as the Markov decision processes (MDPs) [Feinberg, 1996]. When the state and action spaces are finite, many works have established the minimax (near) optimal regret guarantees for MDPs with finite horizon [Azar et al., 2017] and MDPs with infinite horizon [Tossou et al., 2019, He et al., 2021b]. In real applications of RL, however, the state and action spaces may be arbitrarily large and even infinite, which may lead to the curse of dimensionality. To tackle this issue, a common approach is function approximation, which
approximates the value functions of given policies with the leverage of feature mappings. Assuming that the feature mapping which embeds the state-action pairs to a low dimensional embedding space is known, RL with linear function approximation has been well studied recently. In particular, linear mixture MDPs [Ayoub et al., 2020] and linear MDPs [Jin et al., 2020b] are the two models of RL with linear function approximation that have been extensively studied. Notably, their (near) optimal regret guarantees are established by Zhou et al. [2021] and He et al. [2023] respectively. Nevertheless, in scenarios with complex and large-scale data, attaining the true underlying feature mappings might be unrealistic, and thus representation learning is needed. Many empirical works have shown that representation learning can accelerate the sample and computation efficiency of RL [Silver et al., 2018, Laskin et al., 2020, Yang and Nachum, 2021, Stooke et al., 2021, Schwarzer et al., 2021, Xie et al., 2022]. However, representation learning is provably more difficult in RL and other sequential decision-making problems than in non-sequential and non-interactive problems (e.g., supervised learning) [Du et al., 2020, Wang et al., 2021a, Weisz et al., 2021, Uehara et al., 2022]. To pursue sample-efficient RL in the presence of representation learning, recent works have made initial attempts to study the theoretical guarantees of representation learning in RL under the fixed or stochastic loss functions [Uehara et al., 2022, Zhang et al., 2022].
In practice, however, it might be stringent to assume that the loss functions are stochastic. To tackle this issue, Even-Dar et al. [2009] and Yu et al. [2009] propose the first algorithms with provably theoretical guarantees that can handle adversarial MDPs, where the loss functions may change adversarily in each episode. Subsequently, most of the works in this line of research focus on learning tabular MDPs with adversarial loss functions [Neu et al., 2010a,b, 2012, Arora et al., 2012, Zimin and Neu, 2013, Dekel and Hazan, 2013, Dick et al., 2014, Rosenberg and Mansour, 2019a,b, Jin and Luo, 2020, Jin et al., 2020a, Shani et al., 2020b, Chen et al., 2021, Ghasemi et al., 2021, Rosenberg and Mansour, 2021, Jin et al., 2021b, Dai et al., 2022, Chen et al., 2022]. To learn adversarial MDPs with large state and action spaces, some recent works study RL with adversarial loss functions and linear function approximation [Cai et al., 2020, Neu and Olkhovskaya, 2021, Luo et al., 2021b,a, He et al., 2022, Zhao et al., 2023, Kong et al., 2023]. However, all these existing works assume that the state-action feature representations are known. As aforementioned, in complex and high-dimensional environments, the application of these algorithms may be still hindered due to the potential difficulty of knowing the true feature mappings a priori. Therefore, the following question naturally remains open:
Can we devise an algorithm to simultaneously tackle the representation learning and adversarially changed loss functions in reinforcement learning?
In this work, we give an affirmative answer to the above question in the setting of adversarial low-rank MDPs with full-information feedback. Specifically, in this problem, the unknown transition probability kernel admits a low-rank matrix decomposition but the true representations regarding the transitions are not known a priori. Meanwhile, the loss functions are arbitrarily chosen by an adversary across episodes and are revealed to the learner after each episode.
To solve this problem, we propose a policy optimization-based algorithm, which we call Policy Optimization for LOw-rank MDPs (POLO). Specifically, our POLO algorithm obtains an  $\widetilde{O}(K^{5/6}A^{1/2}d\ln(1 + M) / (1 - \gamma)^2)$  regret guarantee for adversarial low-rank MDPs in the full-information feedback setting and is oracle-efficient. Algorithmically, POLO follows similar ideas of optimistic policy optimization methods in that it first constructs optimistic value function estimates and then runs online mirror descent (OMD) over the optimistic value estimates to deal with the adversarially changed loss functions [Shani et al., 2020b, Cai et al., 2020, He et al., 2022, Chen et al., 2022]. However, in the presence of representation learning, the exploration and exploitation needed to learn the adversarial MDPs are more difficult than them in the tabular case [Shani et al., 2020b, Chen et al., 2022] and in the linear case [Cai et al., 2020, He et al., 2022], where the exploration and exploitation can be essentially well balanced by directly modifying the algorithms studying stochastic MDPs in the tabular and linear cases. Concretely, to learn stochastic low-rank MDPs, previous works perform maximum likelihood estimation (MLE) over the experienced transitions [Agarwal et al., 2020, Uehara et al., 2022, Zhang et al., 2022]. Though the balance of representation learning, exploration, and exploitation can be simultaneously handled by the algorithms in these works, these algorithms intrinsically have no regret guarantees but only sample complexity guarantees even in the stochastic case, since these algorithms need to take actions uniformly at certain steps
Table 1: Comparisons of regret bounds with most related works studying adversarial RL with function approximation under unknown transitions.  $K$  is the number of episodes,  $d$  is the ambient dimension of the feature mapping,  $\gamma$  is the discounted factor for infinite-horizon MDPs, and  $S$ ,  $A$ , and  $M$  are the cardinality of the state space, action space, and model class, respectively. Note that the dependence on  $\gamma$  is not strictly comparable since some works originally studying finite-horizon MDPs and these results are translated into results for infinite-horizon MDPs by substituting horizon length  $H$  with  $\Theta(1/(1 - \gamma))$ . The column of "unknown features" indicates whether the algorithm can work in the case when no true feature mappings are known a priori.
Algorithm	Model	Feedback	Regret	Unknown Features
OPPO [Cai et al., 2020]	Linear Mixture MDPs	Full-information	\(\widetilde{O}\left(d\sqrt{K}/(1-\gamma)^{2}\right)\)	x
POWERS [He et al., 2022]	Linear Mixture MDPs	Full-information	\(\widetilde{O}\left(d\sqrt{K}/(1-\gamma)^{3/2}\right)\)	x
LSUOB-REPS Zhao et al. [2023]	Linear Mixture MDPs	Bandit Feedback	\(\widetilde{O}\left(dS^{2}\sqrt{K}+\sqrt{\frac{SAK}{(1-\gamma)}}\right)\)	x
Luo et al. [2021a]	Linear MDPs	Bandit Feedback	\(\widetilde{O}\left(d^{2}K^{14/15}/(1-\gamma)^{4}\right)\)	x
Dai et al. [2023]	Linear MDPs	Bandit Feedback	\(\widetilde{O}\left(\frac{A^{1/9}d^{2/3}K^{8/9}}{(1-\gamma)^{20/9}}\right)\)	x
PO-LSBE Sherman et al. [2023]	Linear MDPs	Bandit Feedback	\(\widetilde{O}\left(\frac{dK^{6/7}}{(1-\gamma)^{2}}+\frac{d^{3/2}K^{5/7}}{(1-\gamma)^{4}}\right)\)	x
GLAP Kong et al. [2023]	Linear MDPs	Bandit Feedback	\(\widetilde{O}\left(d^{7/5}H^{12/5}K^{4/5}\right)\)	x
OPPO+ Zhong and Zhang [2023]	Linear MDPs	Full-information	\(\widetilde{O}\left(\frac{d^{3/4}K^{3/4}+d^{5/2}\sqrt{K}}{(1-\gamma)^{2}}\right)\)	x
POLO (Ours)	Low-rank MDPs	Full-information	\(\widetilde{O}\left(\frac{K^{5/6}A^{1/2}d\ln(1+M)}{(1-\gamma)^{2}}\right)\)Ω\(\left(\frac{\gamma^{2}}{1-\gamma}\sqrt{dAK}\right)\)	✓
in each episode (cf., Lemma 9 of Uehara et al. [2022]).<sup>2</sup> Hence, a straightforward adaptation of their methods from the stochastic setting to the adversarial setting will fail to learn adversarial low-rank MDPs. To cope with this issue, we carefully devise an algorithm with a doubled exploration and exploitation scheme, which interleaves (a) the exploration over transitions required in representation learning; and (b) the exploration and exploitation suggested by the policy optimization. To this end, our algorithm adopts a mixed roll-out policy, which consists of a uniformly explorative policy and a policy optimized by OMD. Through carefully tuning the hyper-parameter of the mixing coefficient used in our mixed policy, we can avoid pulling actions uniformly at random to conduct exploration in each episode and only conduct uniform exploration at a certain fraction of all the episodes (see Section 3.1 for details). Besides, unlike tabular and linear (mixture) MDPs, it is in general hard to achieve the point-wise optimism for each state-action pair. Therefore, depart from previous methods [Shani et al., 2020b, Cai et al., 2020, He et al., 2022] conducting policy optimization in the true model (i.e., the transition kernel characterized by the true representation), our algorithm conducts policy optimization in the fixed learned model with the epoch-based model update, which enables a new analysis scheme that only requires a near optimism at the initial state  $s_0$  (see Section 3.2 for
2With the leverage of the common explore-then-commit (ETC) style conversion, the modified versions of these algorithms can obtain sublinear regret in the setting of low-rank MDPs with stochastic loss functions, but this conversion is still not able to deal with adversarial loss functions.
details). Also, we prove a regret lower bound of order  $\Omega\left(\frac{\gamma^2}{1 - \gamma}\sqrt{dAK}\right)$  for low-rank MDPs with fixed loss functions, which thus also serves as a regret lower bound for our problem and indicates that low-rank MDPs are statistically more difficult to learn than linear MDPs in the regret minimization setting. To the best of our knowledge, this work makes the first step to establish an algorithm with a sublinear regret guarantee for adversarial low-rank MDPs, which permits RL with both nonlinear function approximation and adversarial loss functions. The concrete comparisons between the results of this work and those of previous works are summarized in Table 1.
1.1 Additional Related Works
RL with Function Approximation Significant advances have emerged in RL with function approximation to cope with the curse of dimensionality in arbitrarily large state space or action space. In general, these results fall into two categories. The first category studies RL with linear function approximation, including linear MDPs [Yang and Wang, 2019, Jin et al., 2020b, Du et al., 2020, Zanette et al., 2020, Wang et al., 2020, 2021b, He et al., 2021a, Hu et al., 2022, He et al., 2023] and linear mixture MDPs [Ayoub et al., 2020, Zhang et al., 2021, Zhou et al., 2021, He et al., 2021a, Zhou and Gu, 2022, Wu et al., 2022, Min et al., 2022, Zhao et al., 2023]. Remarkably, He et al. [2023] and Zhou et al. [2021] obtain the nearly minimax optimal regret  $\tilde{O}(d\sqrt{H^3K})$  in linear MDPs and linear mixture MDPs respectively when the loss functions are fixed or stochastic. The other category studies RL with general function approximation. Amongst these works, Jiang et al. [2017], Dann et al. [2018], Sun et al. [2019], Du et al. [2019], and Jin et al. [2021a] study the MDPs satisfying the low Bellman-rank assumption, which assumes the Bellman error matrix has a low-rank factorization. Also, Du et al. [2021] consider a similar but slightly more general assumption termed as bounded bilinear rank. Besides, Russo and Roy [2013], Wang et al. [2020], Jin et al. [2021a], and Ishfaq et al. [2021] study low Eluder dimension assumption, which is originally proposed to characterize the complexity of function classes for bandit problems.
Representation learning in RL arises when the feature mapping that embeds the state-action pairs in RL with linear function approximation is no longer known a priori. Such a problem is typically studied in the setting of low-rank MDPs, which does not assume the feature mapping of state-action pairs is known. Consequently, the setting of low-rank MDPs strictly generalizes the setting of linear MDPs, but at the cost of being more difficult to learn due to potential nonlinear function approximation induced by representation learning. In this line of research, algorithms with provably sample complexity guarantees have been developed in both model-based methods [Agarwal et al., 2020, Ren et al., 2022, Uehara et al., 2022] and model-free methods [Modi et al., 2021, Zhang et al., 2022], respectively. The model-based algorithms of Agarwal et al. [2020], Ren et al. [2022] and Uehara et al. [2022] learn the representation from a given model class of transition probability kernels. In contrast, the model-free methods do not require model learning but may bear some limitations. In particular, Modi et al. [2021] assume the MDPs satisfying the minimal reachability assumption, and the sample complexity of the algorithm of Zhang et al. [2022] only holds for a special class of low-rank MDPs called block MDPs. Besides, representation learning in Markov games has also been investigated recently [Ni et al., 2022].
RL with Adversarial Losses Recent years have witnessed significant advances in learning RL with adversarial losses in the tabular case [Neu et al., 2010a,b, 2012, Arora et al., 2012, Zimin and Neu, 2013, Dekel and Hazan, 2013, Dick et al., 2014, Rosenberg and Mansour, 2019a,b, Jin and Luo, 2020, Jin et al., 2020a, Shani et al., 2020b, Chen et al., 2021, Ghasemi et al., 2021, Rosenberg and Mansour, 2021, Jin et al., 2021b, Dai et al., 2022, Chen et al., 2022]. When it comes to the setting of linear function approximation, various policy optimization-based methods have been established to solve adversarial linear mixture MDPs [Cai et al., 2020, He et al., 2022] and adversarial linear MDPs [Luo et al., 2021a,b, Dai et al., 2023, Sherman et al., 2023, Zhong and Zhang, 2023]. The other line of works studies RL with linear function approximation and adversarial losses using occupancy measure-based methods [Neu and Olkhovskaya, 2021, Zhao et al., 2023, Kong et al., 2023]. To the best of our knowledge, however, there are no works in existing literature studying RL with both nonlinear function approximation and adversarial loss functions.
We consider episodic infinite horizon low-rank MDPs with adversarial loss functions, the preliminaries of which are introduced as follows.
Episodic Infinite-horizon Adversarial MDPs An episodic infinite horizon adversarial MDP is denoted by a tuple  $(\mathcal{S},\mathcal{A},P^{\star},\{\ell_k\}_{k = 1}^K,\gamma ,d_0)$ , where  $\mathcal{S}$  is the state space (with potentially infinitely many states),  $\mathcal{A}$  is the finite action space with cardinality  $|\mathcal{A}| = A$ ,  $P^{\star}:S\times \mathcal{A}\times S\to [0,1]$  is the transition probability kernel such that  $P^{\star}(s^{\prime}\mid s,a)$  is the probability of transferring to state  $s^\prime$  from state  $s$  after executing action  $a$ ,  $\gamma \in [0,1)$  is the discount factor,  $d_0\in \Delta (\mathcal{S})$  is the initial distribution over the state space, and  $\ell_{k}:S\times \mathcal{A}\rightarrow [0,1]$  is the loss function of episode  $k$  chosen by the adversary. For the ease of exposition, we assume  $d_0$  is known.
In this work, we consider a special class of MDPs called low-rank MDPs [Agarwal et al., 2020, Uehara et al., 2022, Zhang et al., 2022]. Specifically, instead of assuming a known true feature mapping, low-rank MDPs only assume that the transition probability kernel  $P^{\star}$  admits a low-rank decomposition, with the formal definition given as follows.
Definition 2.1 (Low-rank MDPs). An MDP is a low-rank MDP if there exist two feature embedding functions  $\phi^{\star}:\mathcal{S}\times \mathcal{A}\to \mathbb{R}^{d}$  and  $\mu^{\star}:\mathcal{S}\rightarrow \mathbb{R}^{d}$  such that for any  $(s,a,s^{\prime})\in \mathcal{S}\times \mathcal{A}\times \mathcal{S},$ $P^{\star}(s^{\prime}\mid s,a) = \mu^{\star}(s^{\prime})^{\top}\phi^{\star}(s,a)$ , where  $\| \phi^{\star}(s,a)\|_{2}\leq 1$  and for any function  $g:\mathcal{S}\rightarrow [0,1],\left\| \int \mu^{\star}(s)g(s)\mathrm{d}(s)\right\|_{2}\leq \sqrt{d}.$
Note that the regularity assumption imposed over  $\phi^{\star}$  and  $\mu^{\star}$  is only for the purpose of normalization.
Function Approximation When the state space is arbitrarily large, function approximation is usually considered to permit sample-efficient learning for MDPs. Since the true feature mapping of state-action pairs is not known a priori in the low-rank MDPs, to make this problem tractable, we assume the access to a realizable model class as previous works [Agarwal et al., 2020, Uehara et al., 2022], detailed in the following.
Assumption 2.1. There exists a known model class  $\mathcal{M} = \{(\mu, \phi) : \mu \in \Psi, \phi \in \Phi\}$  satisfying (a)  $\| \phi(s, a) \|_2 \leq 1$  and  $\int \mu^\top(s') \phi(s, a) \mathrm{d}(s') = 1$  for any  $(s, a, s') \in S \times \mathcal{A} \times S$ ,  $\mu \in \Psi$ ,  $\phi \in \Phi$ ; and (b)  $\left\| \int \mu(s) g(s) \mathrm{d}(s) \right\|_2 \leq \sqrt{d}$  for any function  $g: S \to [0,1]$ . Moreover, it holds that  $\mu^{\star} \in \Psi$  and  $\phi^{\star} \in \Phi$ .
Throughout this paper, for the sake of brevity, we assume that the cardinality of  $\Psi$  and  $\Phi$  are finite, meaning that  $\mathcal{M}$  also has bounded cardinality  $M = |\mathcal{M}|$ . However, note that extending the analyses to the function classes with infinite cardinality but bounded statistical complexity (e.g., VC dimension) is not technically difficult.
Interaction Protocol We now introduce the interaction protocol between the learner and the environment. To begin with, denote by  $d_P^\pi(s,a) = (1 - \gamma)\sum_{\tau=0}^\infty \gamma^\tau d_{P,\tau}^\pi(s,a)$  the state-action occupancy distribution, where  $d_{P,\tau}^\pi(s,a)$  is the probability of visiting  $(s,a)$  at step  $\tau$  under policy  $\pi$  and transition  $P$ . With a slight abuse of notation, let  $d_P^\pi(s) = \sum_{a\in\mathcal{A}}d_P^\pi(s,a)$  be the state occupancy distribution, denoting the probability of visiting state  $s$  under  $\pi$  and  $P$ .
Ahead of time, an MDP is decided by the environment, and only the state space  $S$  and the action space  $\mathcal{A}$  are revealed to the learner. Meanwhile, the adversary secretly chooses  $K$  loss functions  $\{\ell_k\}_{k=1}^K$ , each of which will be used in one episode. The interaction will proceed in  $K$  episodes. At the beginning of episode  $k$ , the learner chooses a stochastic policy  $\pi_k: S \times \mathcal{A} \to [0,1]$ , where  $\pi_k(a \mid s)$  is probability of taking  $a$  at state  $s$ . Starting from an initial state  $s_0 \sim d_0$ , the learner repeatedly executes policy  $\pi_k$  until reaching the termination. After episode  $k$  is terminated, the learner observes a trajectory  $\{(s_{k,\tau},a_{k,\tau})\}_{\tau}$  as well as the loss function  $\ell_k$ .
For each episode  $k$  and each state-action pair  $(s,a)$ , the state-action value  $Q_{k}^{\pi}(s,a)$  is defined as  $Q_{k}^{\pi}(s,a) = \mathbb{E}\left[\sum_{\tau=0}^{\infty} \gamma^{\tau} \ell_{k}(s_{k,\tau},a_{k,\tau}) \mid \pi, P^{\star}, (s_{k,0},a_{k,0}) = (s,a)\right]$ . Also define  $V_{k}^{\pi}(s) = \mathbb{E}_{a \sim \pi(\cdot|s)}[Q_{k}^{\pi}(s,a)]$  and  $V_{k}^{\pi} = \mathbb{E}_{s_{0} \sim d_{0}}[V_{k}^{\pi}(s_{0})]$ . The learning objective is to minimize the pseudo regret with respect to  $\pi^{\star}$ , defined as
$$
\mathcal {R} _ {K} = \mathbb {E} \left[ \sum_ {k = 1} ^ {K} \left(V _ {k} ^ {\pi_ {k}} - V _ {k} ^ {\pi^ {*}}\right) \right],
$$
3Though we focus on episodic infinite-horizon MDPs in this work, note that it is not technically difficult to extend the analyses in this work to the case of episodic finite-horizon MDPs.
where the expectation is taken over the potential randomness of the algorithm,  $\pi^{\star} \in \arg \min_{\pi \in \Pi} \sum_{k=1}^{K} V_k^\pi$  is the fixed optimal policy in hindsight and  $\Pi$  is the set of stochastic policies.
This section presents our POLO algorithm with the pseudocode illustrated in Algorithm 1. At a high level, POLO leverages a mixed roll-out policy to conduct doubled exploration and exploitation, i.e., (a) the exploration over transitions required by representation learning; and (b) the exploration and exploitation over adversarially changed loss functions required by the policy optimization (Section 3.1). To deal with the issue that only the near optimism at the initial state  $s_0$  is available in low-rank MDPs, POLO conducts policy optimization in fixed learned models with the epoch-based model update, which enables a new analysis scheme (Section 3.2).
3.1 Doubled Exploration and Exploitation
Let  $\tilde{\pi}_k$  be the policy of episode  $k$  computed by policy optimization (cf., Eq. (3)). At the beginning of episode  $k$ , our algorithm first collects a state  $s_k \sim d_{P^{\star}}^{\tilde{\pi}_k}$  (Line 6) by invoking a geometric sampling roll-in procedure [Kakade and Langford, 2002, Agarwal et al., 2021, Uehara et al., 2022]. Starting from an initial state  $s_0 \sim d_0$ , at each step  $\tau$ , this roll-in procedure will terminate and return state  $s_\tau$  with probability  $1 - \gamma$ , and otherwise will take action  $a_\tau \sim \tilde{\pi}_k(\cdot \mid s_\tau)$  and transfer to the next state  $s_\tau \sim P^{\star}(\cdot \mid s_\tau, a_\tau)$ .