Title: Learning Adversarial Low-rank Markov Decision Processes with Unknown Transition and Full-information Feedback
Venue: NeurIPS 2023
Accepted: yes
Keywords: adversarial low-rank mdps
Abstract: In this work, we study the low-rank MDPs with adversarially changed losses in the full-information feedback setting. In particular, the unknown transition probability kernel admits a low-rank matrix decomposition \citep{REPUCB22}, and the loss functions may change adversarially but are revealed to the learner at the end of each episode. We propose a policy optimization-based algorithm POLO, and we prove that it attains the $\widetilde{O}(K^{\frac{5}{6}}A^{\frac{1}{2}}d\ln(1+M)/(1-\gamma)^2)$ regret guarantee, where $d$ is rank of the transition kernel (and hence the dimension of the unknown representations), $A$ is the cardinality of the action space, $M$ is the cardinality of the model class that contains all the plausible representations, and $\gamma$ is the discounted factor. Notably, our algorithm is oracle-efficient and has a regret guarantee with no dependence on the size of potentially arbitrarily large state space. Furthermore, we also prove an $\Omega(\frac{\gamma^2}{1-\gamma} \sqrt{d A K})$ regret lower bound for this problem, showing that low-rank MDPs are statistically more difficult to learn than linear MDPs in the regret minimization setting. To the best of our knowledge, we present the first algorithm that interleaves representation learning, exploration, and exploitation to achieve the sublinear regret guarantee for RL with nonlinear function approximation and adversarial losses.

Learning Adversarial Low-rank Markov Decision Processes with Unknown Transition and Full-information Feedback

Abstract

In this work, we study the low-rank MDPs with adversarially changed losses in the full-information feedback setting. In particular, the unknown transition probability kernel admits a low-rank matrix decomposition [Uehara et al., 2022], and the loss functions may change adversarially but are revealed to the learner at the end of each episode. We propose a policy optimization-based algorithm POLO, and we prove that it attains the ˜ O ( K 5 / 6 A 1 / 2 d ln(1 + M ) / (1 -γ ) 2 ) regret guarantee, where d is rank of the transition kernel (and hence the dimension of the unknown representations), A is the cardinality of the action space, M is the cardinality of the model class that contains all the plausible representations, and γ is the discounted factor. Notably, our algorithm is oracle-efficient and has a regret guarantee with no dependence on the size of potentially arbitrarily large state space. Furthermore, we also prove an Ω( γ 2 1 -γ √ dAK ) regret lower bound for this problem, showing that low-rank MDPs are statistically more difficult to learn than linear MDPs in the regret minimization setting. To the best of our knowledge, we present the first algorithm that interleaves representation learning, exploration, and exploitation to achieve the sublinear regret guarantee for RL with nonlinear function approximation and adversarial losses.

1 Introduction

In reinforcement learning (RL), the goal is to learn a (near) optimal policy through the interactions between the learner and the environment, which is typically modeled as the Markov decision processes (MDPs) [Feinberg, 1996]. When the state and action spaces are finite, many works have established the minimax (near) optimal regret guarantees for MDPs with finite horizon [Azar et al., 2017] and MDPs with infinite horizon [Tossou et al., 2019, He et al., 2021b]. In real applications of RL, however, the state and action spaces may be arbitrarily large and even infinite, which may lead to the curse of dimensionality. To tackle this issue, a common approach is function approximation , which

approximates the value functions of given policies with the leverage of feature mappings. Assuming that the feature mapping which embeds the state-action pairs to a low dimensional embedding space is known, RL with linear function approximation has been well studied recently. In particular, linear mixture MDPs [Ayoub et al., 2020] and linear MDPs [Jin et al., 2020b] are the two models of RL with linear function approximation that have been extensively studied. Notably, their (near) optimal regret guarantees are established by Zhou et al. [2021] and He et al. [2023] respectively. Nevertheless, in scenarios with complex and large-scale data, attaining the true underlying feature mappings might be unrealistic, and thus representation learning is needed. Many empirical works have shown that representation learning can accelerate the sample and computation efficiency of RL [Silver et al., 2018, Laskin et al., 2020, Yang and Nachum, 2021, Stooke et al., 2021, Schwarzer et al., 2021, Xie et al., 2022]. However, representation learning is provably more difficult in RL and other sequential decision-making problems than in non-sequential and non-interactive problems ( e.g. , supervised learning) [Du et al., 2020, Wang et al., 2021a, Weisz et al., 2021, Uehara et al., 2022]. To pursue sample-efficient RL in the presence of representation learning, recent works have made initial attempts to study the theoretical guarantees of representation learning in RL under the fixed or stochastic loss functions [Uehara et al., 2022, Zhang et al., 2022].

In practice, however, it might be stringent to assume that the loss functions are stochastic. To tackle this issue, Even-Dar et al. [2009] and Yu et al. [2009] propose the first algorithms with provably theoretical guarantees that can handle adversarial MDPs, where the loss functions may change adversarially in each episode. Subsequently, most of the works in this line of research focus on learning tabular MDPs with adversarial loss functions [Neu et al., 2010a,b, 2012, Arora et al., 2012, Zimin and Neu, 2013, Dekel and Hazan, 2013, Dick et al., 2014, Rosenberg and Mansour, 2019a,b, Jin and Luo, 2020, Jin et al., 2020a, Shani et al., 2020b, Chen et al., 2021, Ghasemi et al., 2021, Rosenberg and Mansour, 2021, Jin et al., 2021b, Dai et al., 2022, Chen et al., 2022]. To learn adversarial MDPs with large state and action spaces, some recent works study RL with adversarial loss functions and linear function approximation [Cai et al., 2020, Neu and Olkhovskaya, 2021, Luo et al., 2021b,a, He et al., 2022, Zhao et al., 2023, Kong et al., 2023]. However, all these existing works assume that the state-action feature representations are known. As aforementioned, in complex and high-dimensional environments, the application of these algorithms may be still hindered due to the potential difficulty of knowing the true feature mappings a priori. Therefore, the following question naturally remains open:

Can we devise an algorithm to simultaneously tackle the representation learning and adversarially changed loss functions in reinforcement learning?

In this work, we give an affirmative answer to the above question in the setting of adversarial lowrank MDPs with full-information feedback. Specifically, in this problem, the unknown transition probability kernel admits a low-rank matrix decomposition but the true representations regarding the transitions are not known a priori. Meanwhile, the loss functions are arbitrarily chosen by an adversary across episodes and are revealed to the learner after each episode.

To solve this problem, we propose a policy optimization-based algorithm, which we call P olicy O ptimization for LO w-rank MDPs (POLO). Specifically, our POLO algorithm obtains an ˜ O ( K 5 / 6 A 1 / 2 d ln(1 + M ) / (1 -γ ) 2 ) regret guarantee for adversarial low-rank MDPs in the fullinformation feedback setting and is oracle-efficient. Algorithmically, POLO follows similar ideas of optimistic policy optimization methods in that it first constructs optimistic value function estimates and then runs online mirror descent (OMD) over the optimistic value estimates to deal with the adversarially changed loss functions [Shani et al., 2020b, Cai et al., 2020, He et al., 2022, Chen et al., 2022]. However, in the presence of representation learning, the exploration and exploitation needed to learn the adversarial MDPs are more difficult than them in the tabular case [Shani et al., 2020b, Chen et al., 2022] and in the linear case [Cai et al., 2020, He et al., 2022], where the exploration and exploitation can be essentially well balanced by directly modifying the algorithms studying stochastic MDPs in the tabular and linear cases. Concretely, to learn stochastic low-rank MDPs, previous works perform maximum likelihood estimation (MLE) over the experienced transitions [Agarwal et al., 2020, Uehara et al., 2022, Zhang et al., 2022]. Though the balance of representation learning, exploration, and exploitation can be simultaneously handled by the algorithms in these works, these algorithms intrinsically have no regret guarantees but only sample complexity guarantees even in the stochastic case, since these algorithms need to take actions uniformly at certain steps

in each episode ( cf. , Lemma 9 of Uehara et al. [2022]). 2 Hence, a straightforward adaption of their methods from the stochastic setting to the adversarial setting will fail to learn adversarial low-rank MDPs. To cope with this issue, we carefully devise an algorithm with a doubled exploration and exploitation scheme, which interleaves (a) the exploration over transitions required in representation learning; and (b) the exploration and exploitation suggested by the policy optimization. To this end, our algorithm adopts a mixed roll-out policy, which consists of a uniformly explorative policy and a policy optimized by OMD. Through carefully tuning the hyper-parameter of the mixing coefficient used in our mixed policy, we can avoid pulling actions uniformly at random to conduct exploration in each episode and only conduct uniform exploration at a certain fraction of all the episodes (see Section 3.1 for details). Besides, unlike tabular and linear (mixture) MDPs, it is in general hard to achieve the point-wise optimism for each state-action pair. Therefore, depart from previous methods [Shani et al., 2020b, Cai et al., 2020, He et al., 2022] conducting policy optimization in the true model ( i.e. , the transition kernel characterized by the true representation), our algorithm conducts policy optimization in the fixed learned model with the epoch-based model update, which enables a new analysis scheme that only requires a near optimism at the initial state s 0 (see Section 3.2 for

details). Also, we prove a regret lower bound of order Ω( γ 2 1 -γ √ dAK ) for low-rank MDPs with fixed loss functions, which thus also serves as a regret lower bound for our problem and indicates that low-rank MDPs are statistically more difficult to learn than linear MDPs in the regret minimization setting. To the best of our knowledge, this work makes the first step to establish an algorithm with a sublinear regret guarantee for adversarial low-rank MDPs, which permits RL with both nonlinear function approximation and adversarial loss functions. The concrete comparisons between the results of this work and those of previous works are summarized in Table 1.

1.1 Additional Related Works

RL with Function Approximation Significant advances have emerged in RL with function approximation to cope with the curse of dimensionality in arbitrarily large state space or action space. In general, these results fall into two categories. The first category studies RL with linear function approximation, including linear MDPs [Yang and Wang, 2019, Jin et al., 2020b, Du et al., 2020, Zanette et al., 2020, Wang et al., 2020, 2021b, He et al., 2021a, Hu et al., 2022, He et al., 2023] and linear mixture MDPs [Ayoub et al., 2020, Zhang et al., 2021, Zhou et al., 2021, He et al., 2021a, Zhou and Gu, 2022, Wu et al., 2022, Min et al., 2022, Zhao et al., 2023]. Remarkably, He et al. [2023] and Zhou et al. [2021] obtain the nearly minimax optimal regret ˜ O ( d √ H 3 K ) in linear MDPs and linear mixture MDPs respectively when the loss functions are fixed or stochastic. The other category studies RL with general function approximation. Amongst these works, Jiang et al. [2017], Dann et al. [2018], Sun et al. [2019], Du et al. [2019], and Jin et al. [2021a] study the MDPs satisfying the low Bellman-rank assumption, which assumes the Bellman error matrix has a low-rank factorization. Also, Du et al. [2021] consider a similar but slightly more general assumption termed as bounded bilinear rank. Besides, Russo and Roy [2013], Wang et al. [2020], Jin et al. [2021a], and Ishfaq et al. [2021] study low Eluder dimension assumption, which is originally proposed to characterize the complexity of function classes for bandit problems.

Representation learning in RL arises when the feature mapping that embeds the state-action pairs in RL with linear function approximation is no longer known a priori. Such a problem is typically studied in the setting of low-rank MDPs, which does not assume the feature mapping of state-action pairs is known. Consequently, the setting of low-rank MDPs strictly generalizes the setting of linear MDPs, but at the cost of being more difficult to learn due to potential nonlinear function approximation induced by representation learning. In this line of research, algorithms with provably sample complexity guarantees have been developed in both model-based methods [Agarwal et al., 2020, Ren et al., 2022, Uehara et al., 2022] and model-free methods [Modi et al., 2021, Zhang et al., 2022], respectively. The model-based algorithms of Agarwal et al. [2020], Ren et al. [2022] and Uehara et al. [2022] learn the representation from a given model class of transition probability kernels. In contrast, the model-free methods do not require model learning but may bear some limitations. In particular, Modi et al. [2021] assume the MDPs satisfying the minimal reachability assumption, and the sample complexity of the algorithm of Zhang et al. [2022] only holds for a special class of low-rank MDPs called block MDPs. Besides, representation learning in Markov games has also been investigated recently [Ni et al., 2022].

RL with Adversarial Losses Recent years have witnessed significant advances in learning RL with adversarial losses in the tabular case [Neu et al., 2010a,b, 2012, Arora et al., 2012, Zimin and Neu, 2013, Dekel and Hazan, 2013, Dick et al., 2014, Rosenberg and Mansour, 2019a,b, Jin and Luo, 2020, Jin et al., 2020a, Shani et al., 2020b, Chen et al., 2021, Ghasemi et al., 2021, Rosenberg and Mansour, 2021, Jin et al., 2021b, Dai et al., 2022, Chen et al., 2022]. When it comes to the setting of linear function approximation, various policy optimization-based methods have been established to solve adversarial linear mixture MDPs [Cai et al., 2020, He et al., 2022] and adversarial linear MDPs [Luo et al., 2021a,b, Dai et al., 2023, Sherman et al., 2023, Zhong and Zhang, 2023]. The other line of works studies RL with linear function approximation and adversarial losses using occupancy measure-based methods [Neu and Olkhovskaya, 2021, Zhao et al., 2023, Kong et al., 2023]. To the best of our knowledge, however, there are no works in existing literature studying RL with both nonlinear function approximation and adversarial loss functions.

2 Preliminaries

Weconsider episodic infinite horizon low-rank MDPs with adversarial loss functions, the preliminaries of which are introduced as follows.

Episodic Infinite-horizon Adversarial MDPs An episodic infinite horizon adversarial MDP is denoted by a tuple ( S , A , P ⋆ , { ℓ k } K k =1 , γ, d 0 ) , 3 where S is the state space (with potentially infinitely many states), A is the finite action space with cardinality |A| = A , P ⋆ : S × A × S → [0 , 1] is the transition probability kernel such that P ⋆ ( s ′ | s, a ) is the probability of transferring to state s ′ from state s after executing action a , γ ∈ [0 , 1) is the discount factor, d 0 ∈ ∆( S ) is the initial distribution over the state space, and ℓ k : S ×A → [0 , 1] is the loss function of episode k chosen by the adversary. For the ease of exposition, we assume d 0 is known.

In this work, we consider a special class of MDPs called low-rank MDPs [Agarwal et al., 2020, Uehara et al., 2022, Zhang et al., 2022]. Specifically, instead of assuming a known true feature mapping, low-rank MDPs only assume that the transition probability kernel P ⋆ admits a low-rank decomposition, with the formal definition given as follows.

Definition 2.1 (Low-rank MDPs) . An MDP is a low-rank MDP if there exist two feature embedding functions ϕ ⋆ : S × A → R d and µ ⋆ : S → R d such that for any ( s, a, s ′ ) ∈ S × A × S , P ⋆ ( s ′ | s, a ) = µ ⋆ ( s ′ ) ⊤ ϕ ⋆ ( s, a ) , where ∥ ϕ ⋆ ( s, a ) ∥ 2 ≤ 1 and for any function g : S → [0 , 1] , ∥ ∥ ∫ µ ⋆ ( s ) g ( s )d( s ) ∥ ∥ 2 ≤ √ d .

Note that the regularity assumption imposed over ϕ ⋆ and µ ⋆ is only for the purpose of normalization.

Function Approximation When the state space is arbitrarily large, function approximation is usually considered to permit sample-efficient learning for MDPs. Since the true feature mapping of state-action pairs is not known a priori in the low-rank MDPs, to make this problem tractable, we assume the access to a realizable model class as previous works [Agarwal et al., 2020, Uehara et al., 2022], detailed in the following.

Assumption 2.1. There exists a known model class M = { ( µ, ϕ ) : µ ∈ Ψ , ϕ ∈ Φ } satisfying (a) ∥ ϕ ( s, a ) ∥ 2 ≤ 1 and ∫ µ ⊤ ( s ′ ) ϕ ( s, a )d ( s ′ ) = 1 for any ( s, a, s ′ ) ∈ S × A × S , µ ∈ Ψ , ϕ ∈ Φ ; and (b) ∥ ∥ ∫ µ ( s ) g ( s )d( s ) ∥ ∥ 2 ≤ √ d for any function g : S → [0 , 1] . Moreover, it holds that µ ⋆ ∈ Ψ and ϕ ⋆ ∈ Φ .

Throughout this paper, for the sake of brevity, we assume that the cardinality of Ψ and Φ are finite, meaning that M also has bounded cardinality M = |M| . However, note that extending the analyses to the function classes with infinite cardinality but bounded statistical complexity ( e.g. , VC dimension) is not technically difficult.

Interaction Protocol We now introduce the interaction protocol between the learner and the environment. To begin with, denote by d π P ( s, a ) = (1 -γ ) ∑ ∞ τ =0 γ τ d π P,τ ( s, a ) the state-action occupancy distribution, where d π P,τ ( s, a ) is the probability of visiting ( s, a ) at step τ under policy π and transition P . With a slight abuse of notation, let d π P ( s ) = ∑ a ∈A d π P ( s, a ) be the state occupancy distribution, denoting the probability of visiting state s under π and P .

Ahead of time, an MDP is decided by the environment, and only the state space S and the action space A are revealed to the learner. Meanwhile, the adversary secretly chooses K loss functions { ℓ k } K k =1 , each of which will be used in one episode. The interaction will proceed in K episodes. At the beginning of episode k , the learner chooses a stochastic policy π k : S × A → [0 , 1] , where π k ( a | s ) is probability of taking a at state s . Starting from an initial state s 0 ∼ d 0 , the learner repeatedly executes policy π k until reaching the termination. After episode k is terminated, the learner observes a trajectory { ( s k,τ , a k,τ ) } τ as well as the loss function ℓ k .

For each episode k and each state-action pair ( s, a ) , the state-action value Q π k ( s, a ) is defined as Q π k ( s, a ) = E [ ∑ ∞ τ =0 γ τ ℓ k ( s k,τ , a k,τ ) ∣ ∣ ∣ π, P ⋆ , ( s k, 0 , a k, 0 ) = ( s, a ) ] . Also define V π k ( s ) = E a ∼ π ( ·| s ) [ Q π k ( s, a )] and V π k = E s 0 ∼ d 0 [ V π k ( s 0 )] . The learning objective is to minimize the pseudo regret with respect to π ⋆ , defined as

where the expectation is taken over the potential randomness of the algorithm, π ⋆ ∈ arg min π ∈ Π ∑ K k =1 V π k is the fixed optimal policy in hindsight and Π is the set of stochastic policies.

3 Algorithm

This section presents our POLO algorithm with the pseudocode illustrated in Algorithm 1. At a high level, POLO leverages a mixed roll-out policy to conduct doubled exploration and exploitation, i.e. , (a) the exploration over transitions required by representation learning; and (b) the exploration and exploitation over adversarially changed loss functions required by the policy optimization (Section 3.1). To deal with the issue that only the near optimism at the initial state s 0 is available in low-rank MDPs, POLO conducts policy optimization in fixed learned models with the epoch-based model update, which enables a new analysis scheme (Section 3.2).

3.1 Doubled Exploration and Exploitation

Let ˜ π k be the policy of episode k computed by policy optimization ( cf ., Eq. (3)). At the beginning of episode k , our algorithm first collects a state s k ∼ d ˜ π k P ⋆ (Line 6) by invoking a geometric sampling roll-in procedure [Kakade and Langford, 2002, Agarwal et al., 2021, Uehara et al., 2022]. Starting from an initial state s 0 ∼ d 0 , at each step τ , this roll-in procedure will terminate and return state s τ with probability 1 -γ , and otherwise will take action a τ ∼ ˜ π k ( · | s τ ) and transfer to the next state s τ ∼ P ⋆ ( · | s τ , a τ ) .

Then our algorithm will further interact with the environment in successive two steps after collecting s k ∼ d ˜ π k P ⋆ (Line 7 - Line 11) like previous works studying low-rank MDPs [Agarwal et al., 2020, Uehara et al., 2022, Zhang et al., 2022]. One of the main differences between previous algorithms and ours lies in how to deal with exploration and exploitation when interacting with the environment. In the case of stochastic low-rank MDPs, the key in the analysis is to ensure the (near) optimism of the optimal policy π ⋆ in the learned model, which essentially requires to bound the performance gap ̂ V π ⋆ k -V π ⋆ k by relating it with the model error regarding π ⋆ , i.e. , E ( s,a ) ∼ d π ⋆ P ⋆ [ ∥ ˆ P k ( · | s, a ) -P ⋆ ( · | s, a ) ∥ 1 ] , where ˆ P k is the MLE solution defined in Eq. (1). However, this model error is not directly controllable, as the algorithm does not know the optimal policy π ⋆ and can not collect data following π ⋆ to bound the model error. Fortunately, by empirical process theory [Geer, 2000, Zhang, 2006], the model error E ( s,a ) ∼ ρ k [ ∥ ˆ P k ( · | s, a ) -P ⋆ ( · | s, a ) ∥ 1 ] regarding the executed policies { ˜ π i } k i =1 is bounded, where ρ k ( s, a ) = 1 k ∑ k i =1 d ˜ π i P ⋆ ( s, a ) . Therefore, previous works [Agarwal et al., 2020, Uehara et al., 2022, Zhang et al., 2022] ensure the optimism of the optimal policy π ⋆ by applying importance weighting to change the measure from d π ⋆ P ⋆ to ρ k when bounding the model error. The complication is that to control the ratio π ⋆ ( a | s ) / ˜ π i ( a | s ) for any ( s, a ) ∈ S × A and i ∈ [ K ] in importance weighting, the algorithms in previous works take actions from the uniform distribution U ( A ) over action space A , which intuitively can be seen as conducting exploration over transitions required by representation learning. Consequently, though these algorithms enjoy excellent sample complexities, they intrinsically do not have regret guarantees due to the uniform exploration over action space, even in the stochastic setting. Moreover, to learn adversarial low-rank MDPs, it is required to take actions adaptively according to the observed loss functions in previous episodes instead of uniformly taking actions. To address this 'conflict' so as to learn adversarial low-rank MDPs, we propose to use a mixed roll-out policy to interleave (a) the exploration over transitions required by representation learning; and (b) the exploration and exploitation over the adversarial loss functions by policy optimization, which we call doubled exploration and exploitation and is pivotal to achieving our regret bound as we will shortly see. Formally, our algorithm will conduct the exploration over the transitions with probability ξ and execute policy ˜ π k optimized by OMD with probability 1 -ξ respectively, as shown in Line 7 - Line 11.

After interacting with the environment, the newly collected data in these two steps will be used to update the datasets (Line 14), and the empirical transition ̂ P k will be updated by performing MLE over the updated datasets by solving (Line 16)

where we denote E D [ f ( s, a, s ′ )] = 1 |D| ∑ ( s,a,s ′ ) ∈D f ( s, a, s ′ ) .

Algorithm 1 Policy Optimization for Low-rank MDPs (POLO)

3.2 Policy Optimization in Fixed Learned Models

It remains to compute the policy ˜ π k +1 to be used in the next episode. To this end, we resort to the canonical OMD framework to perform policy optimization like previous methods [Shani et al., 2020a, Cai et al., 2020, He et al., 2022]. However, previous OMD-based policy optimization methods for tabular and linear (mixture) MDPs [Shani et al., 2020a, Cai et al., 2020, He et al., 2022] critically depend on the point-wise optimism for each state-action pair, i.e. , ̂ Q ˜ π k k ( s, a ) ≤ ℓ k ( s, a ) + γ [ P ⋆ ̂ V ˜ π k k ]( s, a ) , to enable the decomposition ( cf. , Lemma 1 by Shani et al. [2020a])

where ̂ V ˜ π k k and ̂ Q ˜ π k k are the state value and state-action value functions of ˜ π k on ( ̂ P k , ℓ k -̂ b k ) with ̂ b k as some bonus function and the expectation is taken over the randomness of sampling a τ ∼ π ⋆ ( · | s τ ) and s τ +1 ∼ P ⋆ ( · | s τ , a τ ) . The summation of the first term in the above display is contributed by competing with the optimal policy π ⋆ in the true model P ⋆ and can be bounded by common OMDanalysis, which thus can be regarded as conducting policy optimization in the true model. The point-wise optimism guarantees that the second term is less than or equal to 0 .

Nevertheless, in low-rank MDPs, due to the unknown representation, it is generally hard to obtain the above point-wise optimism, which leaves the second optimism term unbounded. To cope with this issue, we instead consider the following decomposition:

where the first term is contributed by competing against the optimal policy π ⋆ in the learned model ̂ P k and can be seen as conducting policy optimization in learned models. This decomposition will be amenable as long as we can achieve a near optimism at the initial state s 0 , i.e. , ̂ V π ⋆ k ( s 0 ) -V π ⋆ k ( s 0 ) ≲ 0 , which turns out to be feasible for low-rank MDPs [Uehara et al., 2022]. However, there remains one more caveat. The first term in Eq. (2) is now no longer directly bounded by OMD analysis, due to the local update nature of OMD-based policy optimization at each state and the state occupancy distribution d π ⋆ ̂ P k now varies across different episodes. To address this issue, Algorithm 1 adopts an epoch-based transition update, in which one epoch has L episodes and the model is only updated at the first episode in one epoch (Line 15 - Line 20). 4 Concretely, Algorithm 1 sets ̂ P k = ̂ P k i and ̂ b k = ̂ b k i , where k i is the first episode of the epoch to which the episode k belongs. In this manner, the learned model is fixed in one epoch, and thus the regret of dealing with the adversarial loss functions by competing against the optimal policy π ⋆ can be bounded in one epoch. Subsequently, at the end of episode k , our algorithm first computes the optimistic value estimate ̂ Q ˜ π k k for current policy ˜ π k under ̂ P k together with the bonus-enhanced loss functions ℓ k -̂ b k by policy evaluation (Line 22). Note that this boils down to planning in the setting of linear MDPs for given features in the learned model and this can be done computationally efficiently [Jin et al., 2020b]. Then the policy is updated by solving

where η > 0 is the learning rate to be tuned later and D F ( x, y ) = F ( x ) -F ( y ) -⟨ x -y, ∇ F ( y ) ⟩ is the Bregman divergence induced by the regularizer F . With F ( π ( · | s )) = ∑ a ∈A π ( a | s ) ln π ( a | s ) as the negative entropy, the closed-form solution to the above display is shown in Line 23, which can be regarded as a kind of soft policy improvement.

4 Analysis

4.1 Regret Upper Bound

The regret upper bound of our POLO algorithm for learning adversarial low-rank MDPs is guaranteed by the following theorem.

Theorem 4.1. Suppose K > d 6 A 3 / (1 -γ ) 6 . For any adversarial low-rank MDP satisfying Definition 2.1, by setting the epoch length L = K 1 / 2 A -1 / 2 d -1 ξ (1 -γ ) , learning rate η = (1 -γ ) √ ln A/ (2 L ) , bonus coefficient α k = O ( √ γ ( A/ξ + d 2 ) ln( Mk/δ )) , regularization coefficient λ k = O ( d ln( Mk/δ )) , mixing coefficient ξ = K -1 / 6 A 1 / 2 d/ (1 -γ ) , and δ = 1 /K , then the regret of Algorithm 1 is upper bounded by

Remark 4.1. Ignoring the dependence on all logarithmic factors but M , the regret upper bound can be simplified as ˜ O ( K 5 / 6 A 1 / 2 d ln(1 + M ) / (1 -γ ) 2 ) . Comparing with the regret lower bound Ω( γ 2 1 -γ √ dAK ) in Theorem 4.2, the regret upper bound in Theorem 4.1 matches in A up to a logarithmic factor but looses in factors of K and d . Also, note that when K is large enough such that ξ and L can be chosen as ξ = K -1 / 6 A 1 / 3 d 2 / 3 / (1 -γ ) and L = K 1 / 2 A -1 d -2 ξ (1 -γ ) = K 1 / 3 A -2 / 3 d -4 / 3 ≥ 1 , meaning that K ≥ d 4 A 2 , the regret upper bound can be further optimized to ˜ O ( K 5 / 6 A 1 / 3 d 2 / 3 ln(1 + M ) / (1 -γ ) 2 ) . Note that this does not conflict with the regret lower bound in Section 4.3 since the magnitude of this upper bound is still larger than that of the regret lower bound as long as K ≥ A 1 / 2 d -1 / 2 γ 6 (1 -γ ) 3 .

4.2 Proof of Regret Upper Bound

We now present the proof of Theorem 4.1. To begin with, recall that in each episode k , after state s k is sampled from d ˜ π k P ⋆ , the actual roll-out policy will be π k ( · | s ) = ξ · U ( A ) + (1 -ξ ) · ˜ π k ( · | s ) . Therefore, it holds that

where the inequality is due to that ∑ ∞ τ =0 γ τ ℓ k ( s τ , a τ ) ∈ [0 , 1 / (1 -γ )] holds for any episode k and any trajectory { ( s τ , a τ ) } ∞ τ =0 . We now turn to bound the first term in Eq. (4) by decomposing it into the following three terms

Bounding OMD REGRET TERM The OMD REGRET TERM is contributed by competing against π ⋆ using ˜ π k with ̂ Q ˜ π k k as loss function in the learned model ̂ P k i . This term is thus bounded by standard OMD analysis, detailed in the following lemma.

Bounding OPTIMISM TERM The OPTIMISM TERM is controlled by choosing appropriate bonus coefficient α k . Note that different from tabular and linear cases, the bonus functions and coefficients here are not devised to control the optimism for each state-action pair. Instead, they are devised to provide a (near) optimism only at the initial state s 0 .

Bounding ESTIMATION BIAS TERM It remains to bound the ESTIMATION BIAS TERM, which comes from the difference between the values of running the same policy ˜ π k in the true model ( i.e. , P ⋆ and ℓ k ) and the learned empirical model ( i.e. , ̂ P k and ℓ k -̂ b k ), respectively. This term can be translated into the error between the true model and the learned model using the common simulation lemma, which is thus bounded by the summation of bonus functions. Note that since the empirical features used to construct our bonus functions vary in each episode, we first relate the bonus functions with the fixed true feature ϕ ⋆ using the one-step trick [Uehara et al., 2022, Zhang et al., 2022], and finally bound this term with the leverage of the canonical elliptical potential lemma. The result is shown in the following lemma.

We refer the readers to Appendix A for the proof of the above lemmas. The proof of Theorem 4.1 is now concluded by first combining Eq. (4), Eq. (5), Lemma 4.1, 4.2, and 4.3 and then choosing L = K 1 / 2 A -1 / 2 d -1 ξ (1 -γ ) , ξ = K -1 / 6 A 1 / 2 d/ (1 -γ ) , and δ = 1 /K .

Intuitively, the epoch length L illustrates a trade-off between dealing with the adversarial losses and the representation learning over the unknown transitions. When L is large, there will be fewer restarts in the running of OMD and thus the learner will suffer less regret contributed by dealing with the adversarial losses as shown by Lemma 4.1. In contrast, a smaller L enables more frequent model updates, which leads to more accurate model estimation and less regret contributed by the representation learning as shown by Lemma 4.2 and 4.3.

4.3 Regret Lower Bound

This section presents the regret lower bound for learning adversarial low-rank MDPs with fixed loss functions in Theorem 4.2, which thus also serves as a regret lower bound for learning adversarial low-rank MDPs with full-information feedback.

Theorem 4.2. Suppose d ≥ 8 , S ≥ d +1 , A ≥ d -3 , and K ≥ 2( d -4) A . Then for any algorithm Alg , there exists an episodic infinite-horizon low-rank MDP M Alg with fixed loss function such that the regret for this MDP is lower bounded by Ω( γ 2 1 -γ √ dAK ) .

Proof Sketch. At a high level, we construct dA hard-to-learn low-rank MDP instances, which are difficult to distinguish in KL divergence but have very different optimal policies. In particular, all the constructed low-rank MDP instances have three levels of states, in which the only state in the first level is a fixed initial state and the states in the third level are absorbing states. Moreover, only one unique absorbing state in the third level has the lowest loss, which is termed as the 'good state'. In the constructed low-rank MDP instance M ( i ⋆ ,a ⋆ ) , the learner can only take specific action to transfer to state s 2 ,i ⋆ in the second level and then take the other specific action to transfer to the unique good state. Due to the unknown representations of state-action pairs, the learner needs to distinguish all these dA low-rank MDP instances, which is essentially equivalent to dealing with a bandit problem with dA 'arms'. The detailed proof of Theorem 4.2 is postponed to Appendix B.

Remark 4.2. Theorem 4.2, to the best of our knowledge, provides the first regret lower bound for learning low-rank MDPs with fixed loss functions. We note that this regret lower bound can hold when d ≪ S and d ≪ A , which thus means that this lower bound is non-trivial. Besides, the regret upper bound in our Theorem 4.1 matches the regret lower bound in A up to a logarithmic factor but looses a factor of ˜ O ( K 1 / 3 d 1 / 2 / ((1 -γ ) γ 2 )) . Importantly, compared with the regret upper bound ˜ O ( d √ K/ (1 -γ ) 3 ) of linear MDPs [He et al., 2023] (the finite horizon H is substituted by the effective horizon Θ(1 / (1 -γ )) in our infinite-horizon setting for a fair comparison), the dependence on A in the regret lower bound of low-rank MDP shows a clear separation between low-rank MDPs and linear MDPs, which demonstrates that low-rank MDPs are statistically more difficult to learn than linear MDPs in the regret minimization setting.

5 Conclusions

In this work, we study learning adversarial low-rank MDPs with unknown transition and fullinformation feedback. We prove that our proposed algorithm POLO achieves the ˜ O ( K 5 / 6 A 1 / 2 d ln(1 + M ) / (1 -γ ) 2 ) regret, which is the first sublinear regret guarantee for this challenging problem. The design of our proposed algorithm features (a) a doubled exploration and exploitation scheme to simultaneously learn the transitions and adversarial loss functions; and (b) policy optimization in the fixed learned models with epoch-based model update to enable a new analysis scheme that only requires the near optimism at the initial state instead of the point-wise optimism. Also, we prove an Ω( γ 2 1 -γ √ dAK ) regret lower bound for this problem, serving as the first regret lower bound for learning low-rank MDPs in the regret minimization setting. Besides, there also remain several interesting future directions to be explored. One natural question is whether it is possible to further optimize the dependence of our regret guarantee on the number of episodes K . The other question is how to also learn adversarial low-rank MDPs with only the bandit feedback available. This is also challenging since the current occupancy measure-based methods and policy optimization-based methods tackling adversarial MDPs with bandit feedback both depend on the point-wise optimism provided by the true feature mapping, which seems not feasible in low-rank MDPs. We hope our results may shed light on better understandings of RL with both nonlinear function approximation and adversarial losses and we leave the above extensions as our future works.

Limitations

We note that in general our algorithm is oracle-efficient (given access to the MLE computation oracle in Eq. (1)) but may not be computationally efficient as previous works studying low-rank MDPs [Agarwal et al., 2020, Uehara et al., 2022, Zhang et al., 2022, Ni et al., 2022]. However, we also remark that in practice, these algorithms including ours are computationally feasible since the computation of MLE is only a standard supervised learning problem and can be implemented using gradient descent methods. The other limitation is that throughout this paper, we assume model class M with bounded cardinality M and the regret upper bound of our algorithm has a logarithmic dependence on M . We remark that similar assumptions and dependence have also appeared in previous theoretical works studying RL with general function approximation [Jiang et al., 2017, Sun et al., 2019]. Also, extending the analyses to an infinite hypothesis class is possible if the hypothesis class has bounded statistical complexity [Agarwal et al., 2020].

References

Pieter Abbeel and Andrew Y. Ng. Exploration and apprenticeship learning in reinforcement learning. In Machine Learning, Proceedings of the Twenty-Second International Conference (ICML 2005), Bonn, Germany, August 7-11, 2005 , volume 119 of ACM International Conference Proceeding Series , pages 1-8. ACM, 2005.

Alekh Agarwal, Sham M. Kakade, Akshay Krishnamurthy, and Wen Sun. FLAMBE: structural complexity and representation learning of low rank mdps. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020.

Alekh Agarwal, Sham M. Kakade, Jason D. Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. J. Mach. Learn. Res. , 22: 98:1-98:76, 2021.

Raman Arora, Ofer Dekel, and Ambuj Tewari. Deterministic mdps with adversarial rewards and bandit feedback. In Nando de Freitas and Kevin P. Murphy, editors, Proceedings of the TwentyEighth Conference on Uncertainty in Artificial Intelligence, Catalina Island, CA, USA, August 14-18, 2012 , pages 93-101. AUAI Press, 2012.

Alex Ayoub, Zeyu Jia, Csaba Szepesvári, Mengdi Wang, and Lin Yang. Model-based reinforcement learning with value-targeted regression. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research , pages 463-474. PMLR, 2020.

Mohammad Gheshlaghi Azar, Ian Osband, and Rémi Munos. Minimax regret bounds for reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017 , volume 70 of Proceedings of Machine Learning Research , pages 263-272. PMLR, 2017.

Qi Cai, Zhuoran Yang, Chi Jin, and Zhaoran Wang. Provably efficient exploration in policy optimization. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research , pages 1283-1294. PMLR, 2020.

Liyu Chen, Haipeng Luo, and Chen-Yu Wei. Minimax regret for stochastic shortest path with adversarial costs and known transition. In Conference on Learning Theory, COLT 2021, 15-19 August 2021, Boulder, Colorado, USA , volume 134 of Proceedings of Machine Learning Research , pages 1180-1215. PMLR, 2021.

Liyu Chen, Haipeng Luo, and Aviv Rosenberg. Policy optimization for stochastic shortest path. In Po-Ling Loh and Maxim Raginsky, editors, Conference on Learning Theory, 2-5 July 2022, London, UK , volume 178 of Proceedings of Machine Learning Research , pages 982-1046. PMLR, 2022.

Yuan Cheng, Ruiquan Huang, Yingbin Liang, and Jing Yang. Improved sample complexity for rewardfree reinforcement learning under low-rank mdps. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023.

Yan Dai, Haipeng Luo, and Liyu Chen. Follow-the-perturbed-leader for adversarial markov decision processes with bandit feedback. CoRR , abs/2205.13451, 2022. doi: 10.48550/arXiv.2205.13451.

Yan Dai, Haipeng Luo, Chen-Yu Wei, and Julian Zimmert. Refined regret for adversarial mdps with linear function approximation. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA , volume 202 of Proceedings of Machine Learning Research , pages 6726-6759. PMLR, 2023.

Christoph Dann, Nan Jiang, Akshay Krishnamurthy, Alekh Agarwal, John Langford, and Robert E. Schapire. On oracle-efficient PAC RL with rich observations. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada , pages 1429-1439, 2018.

Ofer Dekel and Elad Hazan. Better rates for any adversarial deterministic MDP. In Proceedings of the 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA, USA, 16-21 June 2013 , volume 28 of JMLR Workshop and Conference Proceedings , pages 675-683. JMLR.org, 2013.

Travis Dick, András György, and Csaba Szepesvári. Online learning in markov decision processes with changing cost sequences. In Proceedings of the 31th International Conference on Machine Learning, ICML 2014, Beijing, China, 21-26 June 2014 , volume 32 of JMLR Workshop and Conference Proceedings , pages 512-520. JMLR.org, 2014.

Omar Darwiche Domingues, Pierre Ménard, Emilie Kaufmann, and Michal Valko. Episodic reinforcement learning in finite mdps: Minimax lower bounds revisited. In Algorithmic Learning Theory, 16-19 March 2021, Virtual Conference, Worldwide , volume 132 of Proceedings of Machine Learning Research , pages 578-598. PMLR, 2021.

Simon S. Du, Akshay Krishnamurthy, Nan Jiang, Alekh Agarwal, Miroslav Dudík, and John Langford. Provably efficient RL with rich observations via latent state decoding. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA , volume 97 of Proceedings of Machine Learning Research , pages 1665-1674. PMLR, 2019.

Simon S. Du, Sham M. Kakade, Ruosong Wang, and Lin F. Yang. Is a good representation sufficient for sample efficient reinforcement learning? In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020.

Simon S. Du, Sham M. Kakade, Jason D. Lee, Shachar Lovett, Gaurav Mahajan, Wen Sun, and Ruosong Wang. Bilinear classes: A structural framework for provable generalization in RL. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , volume 139 of Proceedings of Machine Learning Research , pages 2826-2836. PMLR, 2021.

Eyal Even-Dar, Sham M. Kakade, and Yishay Mansour. Online markov decision processes. Math. Oper. Res. , 34(3):726-736, 2009.

Feinberg. Markov decision processes: Discrete stochastic dynamic programming (martin l. puterman). SIAM Rev. , 38(4):689, 1996.

Aurélien Garivier, Pierre Ménard, and Gilles Stoltz. Explore first, exploit next: The true shape of regret in bandit problems. Math. Oper. Res. , 44(2):377-399, 2019.

Sara A Geer. Empirical Processes in M-estimation , volume 6. Cambridge university press, 2000.

Mahsa Ghasemi, Abolfazl Hashemi, Haris Vikalo, and Ufuk Topcu. No-regret learning with highprobability in adversarial markov decision processes. In Proceedings of the Thirty-Seventh Conference on Uncertainty in Artificial Intelligence, UAI 2021, Virtual Event, 27-30 July 2021 , volume 161 of Proceedings of Machine Learning Research , pages 992-1001. AUAI Press, 2021.

Jiafan He, Dongruo Zhou, and Quanquan Gu. Logarithmic regret for reinforcement learning with linear function approximation. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , volume 139 of Proceedings of Machine Learning Research , pages 4171-4180. PMLR, 2021a.

Jiafan He, Dongruo Zhou, and Quanquan Gu. Nearly minimax optimal reinforcement learning for discounted mdps. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 22288-22300, 2021b.

Jiafan He, Dongruo Zhou, and Quanquan Gu. Near-optimal policy optimization algorithms for learning adversarial linear mixture mdps. In International Conference on Artificial Intelligence and Statistics, AISTATS 2022, 28-30 March 2022, Virtual Event , volume 151 of Proceedings of Machine Learning Research , pages 4259-4280. PMLR, 2022.

Jiafan He, Heyang Zhao, Dongruo Zhou, and Quanquan Gu. Nearly minimax optimal reinforcement learning for linear markov decision processes. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA , volume 202 of Proceedings of Machine Learning Research , pages 12790-12822. PMLR, 2023.

Pihe Hu, Yu Chen, and Longbo Huang. Nearly minimax optimal reinforcement learning with linear function approximation. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , volume 162 of Proceedings of Machine Learning Research , pages 8971-9019. PMLR, 2022.

Haque Ishfaq, Qiwen Cui, Viet Nguyen, Alex Ayoub, Zhuoran Yang, Zhaoran Wang, Doina Precup, and Lin Yang. Randomized exploration in reinforcement learning with general value function approximation. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , volume 139 of Proceedings of Machine Learning Research , pages 4607-4616. PMLR, 2021.

Nan Jiang, Akshay Krishnamurthy, Alekh Agarwal, John Langford, and Robert E. Schapire. Contextual decision processes with low bellman rank are pac-learnable. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017 , volume 70 of Proceedings of Machine Learning Research , pages 1704-1713. PMLR, 2017.

Chi Jin, Tiancheng Jin, Haipeng Luo, Suvrit Sra, and Tiancheng Yu. Learning adversarial markov decision processes with bandit feedback and unknown transition. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research , pages 4860-4869. PMLR, 2020a.

Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael I. Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory, COLT 2020, 9-12 July 2020, Virtual Event [Graz, Austria] , volume 125 of Proceedings of Machine Learning Research , pages 2137-2143. PMLR, 2020b.

Chi Jin, Qinghua Liu, and Sobhan Miryoosefi. Bellman eluder dimension: New rich classes of RL problems, and sample-efficient algorithms. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 13406-13418, 2021a.

Tiancheng Jin and Haipeng Luo. Simultaneously learning stochastic and adversarial episodic mdps with known transition. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020.

Tiancheng Jin, Longbo Huang, and Haipeng Luo. The best of both worlds: stochastic and adversarial episodic mdps with unknown transition. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 20491-20502, 2021b.

Sham M. Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Claude Sammut and Achim G. Hoffmann, editors, Machine Learning, Proceedings of the Nineteenth International Conference (ICML 2002), University of New South Wales, Sydney, Australia, July 8-12, 2002 , pages 267-274. Morgan Kaufmann, 2002.

Fang Kong, Xiangcheng Zhang, Baoxiang Wang, and Shuai Li. Improved regret bounds for linear adversarial mdps via linear optimization. CoRR , abs/2302.06834, 2023.

Michael Laskin, Aravind Srinivas, and Pieter Abbeel. CURL: contrastive unsupervised representations for reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research , pages 5639-5650. PMLR, 2020.

Tor Lattimore and Csaba Szepesvári. Bandit algorithms . Cambridge University Press, 2020.

Haipeng Luo, Chen-Yu Wei, and Chung-Wei Lee. Policy optimization in adversarial mdps: Improved exploration via dilated bonuses. CoRR , abs/2107.08346, 2021a.

Haipeng Luo, Chen-Yu Wei, and Chung-Wei Lee. Policy optimization in adversarial mdps: Improved exploration via dilated bonuses. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 22931-22942, 2021b.

Yifei Min, Jiafan He, Tianhao Wang, and Quanquan Gu. Learning stochastic shortest path with linear function approximation. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , volume 162 of Proceedings of Machine Learning Research , pages 15584-15629. PMLR, 2022.

Aditya Modi, Jinglin Chen, Akshay Krishnamurthy, Nan Jiang, and Alekh Agarwal. Model-free representation learning and exploration in low-rank mdps. CoRR , abs/2102.07035, 2021.

Gergely Neu and Julia Olkhovskaya. Online learning in mdps with linear function approximation and bandit feedback. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 10407-10417, 2021.

Gergely Neu, András György, and Csaba Szepesvári. The online loop-free stochastic shortest-path problem. In Adam Tauman Kalai and Mehryar Mohri, editors, COLT 2010 - The 23rd Conference on Learning Theory, Haifa, Israel, June 27-29, 2010 , pages 231-243. Omnipress, 2010a.

Gergely Neu, András György, Csaba Szepesvári, and András Antos. Online markov decision processes under bandit feedback. In Advances in Neural Information Processing Systems 23: 24th Annual Conference on Neural Information Processing Systems 2010. Proceedings of a meeting held 6-9 December 2010, Vancouver, British Columbia, Canada , pages 1804-1812. Curran Associates, Inc., 2010b.

Gergely Neu, András György, and Csaba Szepesvári. The adversarial stochastic shortest path problem with unknown transition probabilities. In Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2012, La Palma, Canary Islands, Spain, April 21-23, 2012 , volume 22 of JMLR Proceedings , pages 805-813. JMLR.org, 2012.

Chengzhuo Ni, Yuda Song, Xuezhou Zhang, Chi Jin, and Mengdi Wang. Representation learning for general-sum low-rank markov games. CoRR , abs/2210.16976, 2022.

Tongzheng Ren, Tianjun Zhang, Csaba Szepesvári, and Bo Dai. A free lunch from the noise: Provable and practical exploration for representation learning. In Uncertainty in Artificial Intelligence, Proceedings of the Thirty-Eighth Conference on Uncertainty in Artificial Intelligence, UAI 2022, 1-5 August 2022, Eindhoven, The Netherlands , volume 180 of Proceedings of Machine Learning Research , pages 1686-1696. PMLR, 2022.

Aviv Rosenberg and Yishay Mansour. Online stochastic shortest path with bandit feedback and unknown transition function. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada , pages 2209-2218, 2019a.

Aviv Rosenberg and Yishay Mansour. Online convex optimization in adversarial markov decision processes. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA , volume 97 of Proceedings of Machine Learning Research , pages 5478-5486. PMLR, 2019b.

Aviv Rosenberg and Yishay Mansour. Stochastic shortest path with adversarially changing costs. In Zhi-Hua Zhou, editor, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021 , pages 2936-2942. ijcai.org, 2021.

Daniel Russo and Benjamin Van Roy. Eluder dimension and the sample complexity of optimistic exploration. In Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States , pages 2256-2264, 2013.

Max Schwarzer, Nitarshan Rajkumar, Michael Noukhovitch, Ankesh Anand, Laurent Charlin, R. Devon Hjelm, Philip Bachman, and Aaron C. Courville. Pretraining representations for data-efficient reinforcement learning. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 12686-12699, 2021.

Lior Shani, Yonathan Efroni, Aviv Rosenberg, and Shie Mannor. Optimistic policy optimization with bandit feedback. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research , pages 8604-8613. PMLR, 2020a.

Lior Shani, Yonathan Efroni, Aviv Rosenberg, and Shie Mannor. Optimistic policy optimization with bandit feedback. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research , pages 8604-8613. PMLR, 2020b.

Uri Sherman, Tomer Koren, and Yishay Mansour. Improved regret for efficient online reinforcement learning with linear function approximation. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA , volume 202 of Proceedings of Machine Learning Research , pages 31117-31150. PMLR, 2023.

David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science , 362(6419): 1140-1144, 2018.

Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , volume 139 of Proceedings of Machine Learning Research , pages 9870-9879. PMLR, 2021.

Wen Sun, Nan Jiang, Akshay Krishnamurthy, Alekh Agarwal, and John Langford. Model-based RL in contextual decision processes: PAC bounds and exponential improvements over model-free approaches. In Conference on Learning Theory, COLT 2019, 25-28 June 2019, Phoenix, AZ, USA , volume 99 of Proceedings of Machine Learning Research , pages 2898-2933. PMLR, 2019.

Aristide C. Y. Tossou, Debabrota Basu, and Christos Dimitrakakis. Near-optimal optimistic reinforcement learning using empirical bernstein inequalities. CoRR , abs/1905.12425, 2019.

Masatoshi Uehara, Xuezhou Zhang, and Wen Sun. Representation learning for online and offline RL in low-rank mdps. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022.

Ruosong Wang, Ruslan Salakhutdinov, and Lin F. Yang. Reinforcement learning with general value function approximation: Provably efficient approach via bounded eluder dimension. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020.

Ruosong Wang, Dean P. Foster, and Sham M. Kakade. What are the statistical limits of offline RL with linear function approximation? In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021a.

Yining Wang, Ruosong Wang, Simon Shaolei Du, and Akshay Krishnamurthy. Optimism in reinforcement learning with generalized linear function approximation. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021b.

Gellért Weisz, Philip Amortila, and Csaba Szepesvári. Exponential lower bounds for planning in mdps with linearly-realizable optimal action-value functions. In Algorithmic Learning Theory, 16-19 March 2021, Virtual Conference, Worldwide , volume 132 of Proceedings of Machine Learning Research , pages 1237-1264. PMLR, 2021.

Yue Wu, Dongruo Zhou, and Quanquan Gu. Nearly minimax optimal regret for learning infinitehorizon average-reward mdps with linear function approximation. In International Conference on Artificial Intelligence and Statistics, AISTATS 2022, 28-30 March 2022, Virtual Event , volume 151 of Proceedings of Machine Learning Research , pages 3883-3913. PMLR, 2022.

Zhihui Xie, Zichuan Lin, Junyou Li, Shuai Li, and Deheng Ye. Pretraining in deep reinforcement learning: A survey. CoRR , abs/2211.03959, 2022.

Lin Yang and Mengdi Wang. Sample-optimal parametric q-learning using linearly additive features. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA , volume 97 of Proceedings of Machine Learning Research , pages 6995-7004. PMLR, 2019.

Mengjiao Yang and Ofir Nachum. Representation matters: Offline pretraining for sequential decision making. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , volume 139 of Proceedings of Machine Learning Research , pages 11784-11794. PMLR, 2021.

Jia Yuan Yu, Shie Mannor, and Nahum Shimkin. Markov decision processes with arbitrary reward processes. Math. Oper. Res. , 34(3):737-757, 2009.

Andrea Zanette, David Brandfonbrener, Emma Brunskill, Matteo Pirotta, and Alessandro Lazaric. Frequentist regret bounds for randomized least-squares value iteration. In The 23rd International Conference on Artificial Intelligence and Statistics, AISTATS 2020, 26-28 August 2020, Online [Palermo, Sicily, Italy] , volume 108 of Proceedings of Machine Learning Research , pages 19541964. PMLR, 2020.

Andrea Zanette, Ching-An Cheng, and Alekh Agarwal. Cautiously optimistic policy optimization and exploration with linear function approximation. In Conference on Learning Theory, COLT 2021, 15-19 August 2021, Boulder, Colorado, USA , volume 134 of Proceedings of Machine Learning Research , pages 4473-4525. PMLR, 2021.

Tong Zhang. From ε -entropy to kl-entropy: Analysis of minimum information complexity density estimation. The Annals of Statistics , 34(5):2180-2210, 2006.

Xuezhou Zhang, Yuda Song, Masatoshi Uehara, Mengdi Wang, Alekh Agarwal, and Wen Sun. Efficient reinforcement learning in block mdps: A model-free representation learning approach. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , volume 162 of Proceedings of Machine Learning Research , pages 26517-26547. PMLR, 2022.

Zihan Zhang, Jiaqi Yang, Xiangyang Ji, and Simon S. Du. Improved variance-aware confidence sets for linear bandits and linear mixture MDP. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 4342-4355, 2021.

Canzhe Zhao, Ruofeng Yang, Baoxiang Wang, and Shuai Li. Learning adversarial linear mixture markov decision processes with bandit feedback and unknown transition. In The Eleventh International Conference on Learning Representations , 2023.

Han Zhong and Tong Zhang. A theoretical analysis of optimistic proximal policy optimization in linear markov decision processes. arXiv preprint arXiv:2305.08841 , 2023.

Dongruo Zhou and Quanquan Gu. Computationally efficient horizon-free reinforcement learning for linear mixture mdps. CoRR , abs/2205.11507, 2022.

Dongruo Zhou, Quanquan Gu, and Csaba Szepesvári. Nearly minimax optimal reinforcement learning for linear mixture markov decision processes. In Conference on Learning Theory, COLT 2021, 15-19 August 2021, Boulder, Colorado, USA , volume 134 of Proceedings of Machine Learning Research , pages 4532-4576. PMLR, 2021.

Alexander Zimin and Gergely Neu. Online learning in episodic markovian decision processes by relative entropy policy search. In Christopher J. C. Burges, Léon Bottou, Zoubin Ghahramani, and Kilian Q. Weinberger, editors, Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States , pages 1583-1591, 2013.

Peer reviews:
comments: SUMMARY:
The paper studies low-rank MDPs with adversarially changing losses in the full-information feedback setting. They assume the unknown transition probability function admits a low-rank matrix decomposition. They present POLO algorithm, a policy optimization-based algorithm, and prove it has sublinear regret in the number of episodes $K$, but linear dependency in $d$ which is the rank of the transition kernel. The regret is w.r.t the optimal policy for the discounted return value function.
The authors claim they are the first to present and algorithm that interleaves representation learning, exploration, and exploitation to achieve the sublinear regret  guarantee for RL with nonlinear function approximation and adversarial losses.

STRENGTHS:
1.	POLO is the first algorithm that achieves sublinear regret with unknown features (although the setting is full information).
2.	The authors provide a clear comparison between their work to previous works in the area, which emphasis their contribution.
3.     Regret upper bound stated clearly, the analysis sketch is standard and clear (not fully checked).

WEAKNESSES:
1.	The full information feedback is a restrictive and unrealistic assumption.
2.	The extensive literature review in Section 2 can be shorter, and instead use that space to have more proof sketches. That, in my opinion, can benefit the reader.

QUESTIONS:
1.	What is the difference between low-rank MDPs to Generalized Linear Model? As I see it, GLM is more general model. Do your results hold for this setting as well?
2.	Same question for low Bellman rank MDPs, that also seem to generalize low-rank MDPs.
3.	Do you have thoughts about how to remove the full-information assumption?

LIMITATIONS:
n/a
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work focuses on the low-rank MDPs with adversarial losses in the full-information feedback setting. Different from the previous work which assumes known features, this work considers the combination of representation learning and regret minimization problem, and is of the first result under this specific topic. \n\nThe low-rank MDPs are defined in Definition 3.1, which admit certain low-rank feature structure that the transition dynamics can be presented with respect to two feature embedding functions. As described in the interaction protocol, the learner does not have accesses to the feature functions, and is required to learn these functions via the interaction between the environment. \n\nWhile the learner has to collect enough samples to estimate the feature functions, the goal of the learner is to minimize her regret (defined in Line 201), that is, the difference between the cumulative loss suffered, and that of the optimal policy in hindsight. The loss functions are selected adversarially   that can change from episode to episode arbitrarily. \n\nTo tackle these two problems simultaneously, the authors combine the techniques of representation learning and the RL with linear MDPs and adversarial losses together into Algorithm 1. Line 5-10 follow the idea of Uehara et al. [2022] to collect the samples for representation learning, Line 11-17 follow the idea of learning adversarial Linear MDPs like the work of Cai et al. [2020], given the estimated feature functions solved in Eq. (1). \n\nThe result is presented in Theorem 5.1, which guarantees that the regret of Algorithm 1 is bounded by $\\widetilde{\\mathcal{O}}( \\frac{K^{3/4}}{1-\\gamma} + \\frac{\\sqrt{K}}{(1-\\gamma)^2})$, where the first term is attributed to the representation learning of unknown features, and the second term is for the adversarial losses.\n\nSTRENGTHS:\n1. Prior to this work, it is unknown whether the regret minimization problem and the representation learning problem can be dealt simultaneously with adversarial losses. This work is the first one to achieve sub-linear regret, which sheds light on this direction. \n2. Based on the previous works on the representation learning of MDPs (such as Uehara et al. [2022] and Agarwal et al. [2020]), and on the linear MDPs (or low-rank MDPs) with adversarial losses (such as Cai et al. [2020] and Luo et al. [2021a]), the authors combine the techniques of these two fields hierarchically, under a novel scheme `doubled exploration and exploitation', to address the regret minimization problem.\n\nWEAKNESSES:\n1. The designed algorithm suffers from huge computation cost (Line 13, 15, 16, and 17) when the state space is very large, which is the same as the previous algorithm OPPO in the previous work of Cai et al. [2020]. It is not very clear whether this could be improved with any planner oracle.\n\nQUESTIONS:\n1. This result $\\mathcal{O}(K^{3/4})$ does not match the $\\sqrt{K}$ regret lower bound for the adversarial losses. Any specific conjecture on the lower bound/upper bound? \n2. Any conjecture on the bandit feedback setting when limited the MDPs to the linear Mixture MDPS of Zhao et al. [2023]? \n3. The loss function $\\ell_k$ is designed as a function of $ S \\times A \\rightarrow [0,1]$, which does not admit the linear structure. Any idea to improve the result given the fact that $\\ell_k(s,a) = \\theta_k^\\top \\phi^\\star(s,a) $ when $\\theta_k$ is revealed at the end of the episode $k$? \n4. The second term in the regret bound, comparing with the regret bound of OPPO and LSUOB-REPS, shaves a factor of $d$? Could you explain why this holds true, and whether this implies that your algorithm achieves better performance given the feature functions?\n\nLIMITATIONS:\nThis work is pure theoretical, does not have any potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider the problem of learning an adversarial low-rank infinite episodic MDP with unknown transition and full information feedback. \nThe idea behind this problem is that in many RL applications, the state and action spaces might be prohibitively large, rending results that scale with these metrics meaningless. Several approaches have used the assumption that there is a feature mapping state-action pairs to a low dimensional embedding space. While the common practice is to assume that this embedding is known, in the present problem the learner has to learn how the data is represented. \nThe second challenge faced by the learner is that the loss functions change adversarially. These two challenges have been studied separately, but this work is the first to study them simultaneously. \n\nTo do so, they work under the assumption that there exists an unknown low-rank embedding of the state-action pairs and propose an algorithm to tackle this problem: Policy Optimization for LOw rank MDPS (POLO). This algorithm relies on the standard Online Mirror Descent technique, combined with a bonus function which lowers the observed losses to favorize exploration.\n\nThe authors then provide a rigourous analysis of the regret and show that their algorithm reaches sublinear regret. \nThe regret analysis follows a standard  decomposition of the regret into an estimation bias term, the OMD regret term and an optimism term, which respectively compare the true value function with the eztimated value function of the policy played by the learner, the difference between the performance of the policy played by the learner and the best policy in hindsight on the estimated value function, and finally the  difference between the estimated value function and the true value function for the best policy.\n\nSTRENGTHS:\nThis paper tackles a new and challenging problem by combining two notably difficult aspects of online learning with MDPs: handling adversarial losses and having to learn the state-action mapping. Each of these problems has received a lot of attention in the past few years and are really relevant for the online learning  community.\n\nThe authors achieve their goal by presenting an algorithm that adapts to both of these aspects simultaneously and obtains sublinear regret.\nThe algorithm that they present buil;dss upon standard tools in the litterature and has an appreciably clear and compact analysis. \n\nThe authors properly discuss the limitations of their work, notably by questioning the computational efficiency of their approach and ensuring that their algorithm is coomputationally feasible. \n\nOverall, this paper will provide a good baseline for the problems of representation learning of MDPs in adversarial environments.\n\nWEAKNESSES:\nIt would be interesting to provide a the lower bound for the problem: from the simple online learning with full information feedback, having at least $\\sqrt{K}$ regret from playing adversarial losses is unavoidable.  In this work, it is stated that the first term in the regret bound, which scales with $K^{3/4}/(1 - \\gamma)$ comes from learning the unknown transition kernel $P^\\star$. Do you have a lower bound for this part the representation learning part of the problem?\n\nQUESTIONS:\nIs there a lower bound for this problem, and could you elaborate on the cost of learning the representation?\n\nLIMITATIONS:\nThe authors provided an interesting discussion of the potential limitations of their work and addressed them adequately.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies low-rank MDPs with unknown and fixed transition and full-information adversarial losses. The proposed algorithm generalize RepUCB from the fixed reward setting to the adversarial reward setting. The main idea of the algorithm is to replace the greedy policy in RepUCB to an incremental policy update with exponential weights. Also, the proposed algorithm interleaves exploration and exploitation to obtain low regret.\n\nSTRENGTHS:\n- This is an elegant generalization of RepUCB to the adversarial setting. It is also the first paper that simultaneously deals with non-linear function approximation and adversarial losses.\n\nWEAKNESSES:\n- The technical novelty is a bit limited -- Given OPPO and POWERS, it becomes more or less clear that the key technique for unknown transition + full-information loss feedback setting is to ensure optimism and use a no-regret algorithm over the optimistically-biased losses. Though the form of optimism in low-rank MDP (i.e., optimism only in the initial state) is slightly different from those in linear mixture MDP, with full-information loss feedback, the key challenges is already resolved in RepUCB. It is not surprising that most of the analysis follows those of RepUCB except for the OMD regret term. \n- The proposed algorithm performs an epsilon-greedy styled exploration, which leads to a sub-optimal $K^{3/4}$ regret bound. To improve the technical novelty, I think the paper should try to design some less trivial exploration scheme and improve over the $K^{3/4}$ bound. Another direction to improve the technical novelty is to study the bandit feedback case.  \n- The comparison with Rep-UCB around Line 259-268 and the comment that your algorithm is tighter in $d, A, \\gamma$ is slightly off. The reason is that your bound is better than RepUCB only when $K^{3/4}A^{1/2}d/(1-\\gamma) < d^{4/3}A^{2/3}K^{2/3}/(1-\\gamma)^{4/3}$, which is equivalent to $K<d^4A^2/(1-\\gamma)^4$. In this regime, both regret bounds $K^{3/4}A^{1/2}d/(1-\\gamma)$ and $d^{4/3}A^{2/3}K^{2/3}/(1-\\gamma)^{4/3}$ are at least $K$, meaning that this is an *vacuous* regime where both regret bounds are meaningless. In other words, there is no case your regret bound is better than that of RepUCB. If we calculate the sample complexity, the two algorithms would give $\\frac{A^2d^4}{(1-\\gamma)^4\\epsilon^4}$ and $\\frac{A^2d^4}{(1-\\gamma)^4\\epsilon^3}$ respectively, so there is actually no improvement in terms of $d, A, (1-\\gamma)$.\n\nQUESTIONS:\nSee the above sections.\n\nLIMITATIONS:\nThere is no societal limitation.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}