Title: Beating Adversarial Low-Rank MDPs with Unknown Transition and Bandit Feedback
Venue: NeurIPS 2024
Accepted: yes
Keywords: low-rank MDPs

adversarial losses

bandit feedback
Abstract: We consider regret minimization in low-rank MDPs with fixed transition and adversarial losses. Previous work has investigated this problem under either full-information loss feedback with unknown transitions (Zhao et al., 2024), or bandit loss feedback with known transitions (Foster et al., 2022). First, we improve the $poly(d, A, H)T^{5/6}$ regret bound of Zhao et al. (2024) to $poly(d, A, H)T^{2/3}$ for the full-information unknown transition setting, where $d$ is the rank of the transitions, $A$ is the number of actions, $H$ is the horizon length, and $T$ is the number of episodes. Next, we initiate the study on the setting with bandit loss feedback and unknown transitions. Assuming that the loss has a linear structure, we propose both model-based and model-free algorithms achieving $poly(d, A, H)T^{2/3}$ regret, though they are computationally inefficient. We also propose oracle-efficient model-free algorithms with $poly(d, A, H)T^{4/5}$ regret. We show that the linear structure is necessary for the bandit case—without structure on the reward function, the regret has to scale polynomially with the number of states. This is contrary to the full-information case (Zhao et al., 2024), where the regret can be independent of the number of states even for unstructured reward functions.

Beating Adversarial Low-Rank MDPs with Unknown Transition and Bandit Feedback

Abstract

We consider regret minimization in low-rank MDPs with fixed transition and adversarial losses. Previous work has investigated this problem under either fullinformation loss feedback with unknown transitions (Zhao et al., 2024), or bandit loss feedback with known transition (Foster et al., 2022). First, we improve the poly ( d, A, H ) T 5 / 6 regret bound of Zhao et al. (2024) to poly ( d, A, H ) T 2 / 3 for the full-information unknown transition setting, where d is the rank of the transitions, A is the number of actions, H is the horizon length, and T is the number of episodes. Next, we initiate the study on the setting with bandit loss feedback and unknown transitions. Assuming that the loss has a linear structure, we propose both model-based and model-free algorithms achieving poly ( d, A, H ) T 2 / 3 regret, though they are computationally inefficient. We also propose oracle-efficient modelfree algorithms with poly ( d, A, H ) T 4 / 5 regret. We show that the linear structure is necessary for the bandit case-without structure on the reward function, the regret has to scale polynomially with the number of states. This is contrary to the full-information case (Zhao et al., 2024), where the regret can be independent of the number of states even for unstructured reward function.

1 Introduction

We study online reinforcement learning (RL) in low-rank Markov Decision Processes (MDPs). Lowrank MDPs is a class of MDPs where the transition probability can be decomposed as an inner product between two low-dimensional features, i.e., P ( x ′ ∣ x, a ) = ϕ ⋆ ( x, a ) ⊺ µ ⋆ ( x ′ ) , where P ( x ′ ∣ x, a ) is the probability of transitioning to state x ′ when the learner takes action a on state x , and ϕ ⋆ , µ ⋆ are two feature mappings. The ground truth features ϕ ⋆ and µ ⋆ are unknown to the learner. This setting has recently caught theoretical attention due to its simplicity and expressiveness (Agarwal et al., 2020; Uehara et al., 2021; Zhang et al., 2022a; Cheng et al., 2023; Modi et al., 2024; Zhang et al., 2022b; Mhammedi et al., 2024a; Huang et al., 2023). In particular, since the learner does not know the features, it is necessary for the learner to perform feature learning (or representation learning ) to approximate them. This allows low-rank MDPs to model the additional difficulty not present in traditional linear function approximation schemes where the features are given, such as in linear MDPs (Jin et al., 2020b) and in linear mixture MDPs (Ayoub et al., 2020). Since feature learning is an indispensable part of modern deep RL pipelines, low-rank MDP is a model that is closer to practice than traditional linear function approximation.

Zakaria Mhammedi ∗

Most prior theoretical work on low-rank MDPs focuses on reward-free learning; this is a setting where instead of focusing on a particular reward function, the goal is to learn a model for the transitions (or, in the model-free setting, a small set of policies with good state cover), that enables policy optimization for any downstream reward functions. While this is a reasonable setup in some cases, in other applications, the learner can only obtain loss information from interactions with the environment, and only observes the loss on the state-actions that have been visited (i.e., bandit feedback). This introduces additional challenges to the learner.

Furthermore, in many online learning scenarios, the loss function may change over time, reflecting the non-stationary nature of the environment or task switches (Padakandla et al., 2020). This could be modeled by the adversarial MDP setting, where the loss function changes arbitrarily from one episode to the next, and the changes might even depend on the behavior of the learner. This setting is also extensively studied, but mostly restricted to tabular MDPs (Rosenberg and Mansour, 2019; Jin et al., 2020a; Shani et al., 2020; Luo et al., 2021) or traditional linear function approximation schemes (Cai et al., 2020; Luo et al., 2021; He et al., 2022; Zhao et al., 2022; Sherman et al., 2023b; Dai et al., 2023; Liu et al., 2023). The work by Zhao et al. (2024) initiated the study on adversarial MDPs in low-rank MDPs, but their work is restricted to full-information loss feedback.

When feature learning, bandit feedback, and adversarial losses are combined, the problem becomes highly challenging, and to the best of our knowledge their are no provably efficient algorithms to tackle this setting. In this work, we provide the first result for this combination. We hope that our result would bring new ideas to RL in practice, where all three elements are usually present simultaneously. We give several main results, targeting at either tighter regret (i.e., the performance gap between the optimal policy and the learner) or computational efficiency, as summarized in Table 1. Below we give a brief introduction for each of them. A more thorough related work review is in Appendix A.

T 2 / 3 -regret algorithm under full-information feedback (Algorithm 1). This setting is studied by the only prior work in adversarial low-rank MDPs (Zhao et al., 2024), and we greatly improve their T 5 / 6 regret bound to T 2 / 3 . Our algorithm begins with a model-based initial exploration phase to estimate the transition. It then performs policy optimization where the critic is the Q value induced by the estimated transition and the full information loss.

T 2 / 3 -regret model-based/model-free inefficient algorithm under bandit feedback (Algorithm 2, Algorithm 5). Algorithm 2 starts with a model-based initial exploration phase to learn an estimated transition, and then runs exponential weights over policy space for regret minimization in the second phase. To tackle bandit feedback, we construct a novel loss estimator that leverages the structure of low-rank MDP to perform accurate off-policy evaluation. Algorithm 5 starts with a

different exploration phase, where it calls VoX (Mhammedi et al., 2023) to learn a policy cover; VoX is a model-free, reward-free exploration algorithm. After this initial exploratory phase, the algorithm also applies exponential weights and utilizes the same loss estimator as in Algorithm 2. However, due to its model-free nature, certain components of the estimator cannot be directly accessed and must be derived through specific optimizations.

T 4 / 5 -regret model-free oracle-efficient algorithm under bandit feedback (Algorithm 3, Algorithm 4). Algorithm 3 also starts with the model-free exploration algorithm VoX (Mhammedi et al., 2023) to learn a policy cover. After that, the algorithm operates in epochs; during epoch k , the algorithm commits to a fixed mixture of policies. This mixture consists of certain exploratory policies (based on the policy cover from the initial phase) and a policy computed using an online learning algorithm based on estimated Q -functions from previous epochs (these serve as loss functions). Algorithm 4 deals with the much more challenging setting of an adaptive adversary with bandit feedback. Here, we make the additional assumption that the loss feature, which may be different from the feature of the low-rank decomposition, is given. The algorithm is similar to Algorithm 3 with key differences outlined in Section 4.3.

2 Preliminaries

We study the episodic online reinforcement learning setting with horizon H . We consider an MDP M=(X , A , P ⋆ 1 ∶ H ) , where X represents a countable (possibly infinite) state space 2 , A is a finite action space, and P ⋆ h ∶ X × A → ∆ (X) denotes the transition kernel from layer h to h + 1 . We assume that the initial state x 1 ∈ X is fixed for simplicity without loss of generality. For any policy π ∶ X ↦ ∆ (A) and arbitrary set of transition kernels { P h } h ∈[ H ] , we let P P,π denote the law over ( x 1 , a 1 , . . . , x H , a H ) induced by the process of setting x 1 = x 1 , sampling a 1 ∼ π 1 (⋅ ∣ x 1 ) , then for h = 2 , . . . , H , x h ∼ P h -1 (⋅ ∣ x h -1 , a h -1 ) and a h ∼ π h (⋅ ∣ x h ) . We let E P,π denote the corresponding expectations. Further, we let d P,π h ( x ) ∶= P P,π [ x h = x ] denote the occupancy of x ∈ X . We also let d P,π h ( x, a ) ∶= P P,π [ x h = x, a h = a ] . Further, we let E π = E P ⋆ ,π , P π = P P ⋆ ,π , and d π h = d P ⋆ ,π h . We use π ○ h π ′ to denote a policy that follows π k (⋅ ∣ ⋅) for k < h and π ′ k (⋅ ∣ ⋅) for k ≥ h . Similarly, π ○ h π ′ ○ h ′ π ′′ denotes a policy that follows π k for k < h , π ′ k for h ≤ k < h ′ and π ′′ k for k > h ′ .

We consider a learner interacting with the MDP M for T episodes with adversarial loss functions. Before the game starts, an oblivious adversary chooses the loss functions for all episodes ( ℓ t 1 ∶ H ∶ X × A → [ 0 , 1 ]) T t = 1 . For each episode t ∈ [ T ] , the learner starts at state x t 1 = x 1 , then for each step h ∈ [ H ] within episode t , the learner observes state x t h ∈ X h , chooses an action a t h ∈ A , then suffers loss ℓ t h ( x t h , a t h ) . The state x t h + 1 at the next step is drawn from transition P ⋆ h (⋅ ∣ x t h , a t h ) . We consider bandit feedback setting where the learner could only observe the losses ℓ t 1 ( x t 1 , a t 1 ) , . . . , ℓ t H ( x t H , a t H ) at the visited state-action pairs.

We let Π ∶= { π ∶ X → ∆ (A)} denote the set of Markovian policies. For policy π ∈ Π , loss ℓ and transition kernels P 1 ∶ H , we denote by Q P,π h (⋅ , ⋅ ; ℓ ) the state-action value function (a.k.a. Q -function) at step h ∈ [ H ] with respect to the transitions P 1 ∶ H and loss ℓ ; that is for all ( x, a ) ∈ X × A . We let V P,π h ( x ; ℓ ) ∶= max a ∈A Q P,π h ( x, a ; ℓ ) be the corresponding state value function at layer h . Further, we write Q π h (⋅ , ⋅ ; ℓ ) ∶= Q P ⋆ ,π h (⋅ , ⋅ ; ℓ ) and V π h (⋅ ; ℓ ) ∶= V P ⋆ ,π (⋅ ; ℓ ) .

For all of our algorithms except for Algorithm 4, we aim to construct (possibly randomized) policies { π t } t ∈[ T ] that ensure a sublinear pseudo-regret with respect to the best-fixed policy; that is,

ForAlgorithm 4, we bound the standard regret

with high probability. This allows it to handle adaptive adversary.

Throughout, we will assume that the MDP M is low-rank with unknown feature maps ϕ ⋆ h and µ ⋆ h .

Assumption 2.1 (Low-Rank MDP) . There exist (unknown) features maps ϕ ⋆ 1 ∶ H ∶ X × A → R d and µ ⋆ 1 ∶ H ∶ X → R d , such that for all h ∈ [ H -1 ] and ( x, a, x ′ ) ∈ X × A × X :

Furthermore, for all h ∈ [ H ] , the feature maps µ ⋆ h and ϕ ⋆ h are such that sup ( x,a )∈X×A ∥ ϕ ⋆ h ( x, a )∥ ≤ 1 and ∥ ∑ x ∈X g ( x ) ⋅ µ ⋆ h ( x )∥ ≤ √ d , for all g ∶ X → [ 0 , 1 ] .

Loss function under bandit feedback. For bandit feedback setting, we make the following additional linear assumption on the losses; in the sequel, we will argue that this is necessary to avoid a sample complexity scaling with the number of states. This linear loss assumption also appears in Ren et al. (2022); Zhang et al. (2022a) for stochastic low-rank MDPs. Note that for the full-information feedback setting, such an assumption is not required.

Assumption 2.2 (Loss Representation) . For any t ∈ [ T ] and layer h , there is a vector g t h ∈ B d ( 1 ) such that the loss ℓ t h ( x, a ) at round t satisfies:

We note that there is no loss of generality in assuming that the losses are expressed using the same features ϕ ⋆ 1 ∶ H as the low-rank structure in (4). This is because if the losses have different features, we can simply combine these features with the low-rank features, and redefine ϕ ⋆ accordingly. For the bulk of our results (and as stated in the prequel), we will assume that the losses { ℓ t h (⋅ , ⋅)} h ∈[ H ] ,t ∈[ T ] (or equivalently { g t h } h ∈[ H ] ,t ∈[ T ] under Assumption 2.2) are chosen by an aversary before the start of the game (i.e. oblvious adversary). In Section 4.3, we will present a model-free, oracle-efficient algorithm for an adaptive adversary.

Function approximation. So far, Assumption 2.1 and Assumption 2.2 are in line with assumptions made in the linear MDP setting (Jin et al., 2020b). However, unlike in linear MDPs, we do not assume that the feature maps ϕ ⋆ 1 ∶ H are known. To facilitate representation learning and ultimately a sublinear regret, we need to make realizability assumptions. In particular, in the model-free setting, we assume we have a function class Φ that contains the true features ϕ ⋆ 1 ∶ H . In the model-based setting, we additionally assume access to a function class Υ that contains the feature maps µ ⋆ 1 ∶ H 3 . We will formalize these assumptions in their corresponding sections in the sequel.

Other notation. For ψ ∶ Π → R d , we define John ( ψ, Π ) as a distribution µ ∈ ∆ ( Π ) such that ∥ ψ ( π )∥ 2 G -1 ≤ d for all π ∈ Π , where G = ∑ π ∈ Π µ ( π ) ⋅ ψ ( π ) ψ ( π ) ⊺ . This is the standard John's exploration or G -optimal design, which always exists.

3 Model-based Algorithms for Adversarial Low-rank MDPs

In this section, we discuss adversarial low-rank MDPs under model-based assumption. The modelbased assumption is formalized in the Assumption 3.1 below. This assumption is standard which also appears in prior works on model-based learning in low-rank MDPs (Agarwal et al., 2020; Uehara et al., 2021; Zhang et al., 2022a; Cheng et al., 2023; Zhao et al., 2024).

Assumption 3.1 (Model-based assumption) . The learner has access to two model spaces Φ and Υ such that ϕ ⋆ ∈ Φ and µ ⋆ ∈ Υ . Moreover, for any ϕ ∈ Φ , µ ∈ Υ , and h ∈ [ 2 .. H ] , we have sup ( x,a )∈X×A ∥ ϕ h -1 ( x, a )∥ ≤ 1 , ∑ x ′ ∈X ϕ h -1 ( x, a ) ⊺ µ h ( x ′ ) = 1 and ∥ ∑ x ∈X g ( x ) ⋅ µ h ( x )∥ ≤ √ d , for all g ∶ X → [ 0 , 1 ] .

3.1 Adversarial Low-rank MDPs with Full Information

We first discuss learning adversarial low-rank MDPs with full information and model-based assumption. This setting aligns with Zhao et al. (2024), and our Algorithm 1 successfully improves their regret from T 5 / 6 to T 2 / 3 .

Algorithm 1 Model-Based Algorithm for Full-Information Feedback

1: Let η = 1 H √ T , ϵ = ( Hd 2 ∣A∣( d 2 + ∣A∣)) 1 3 T -1 3 , and T 0 = ̃ O( ϵ -2 H 3 d 2 ∣A∣( d 2 + ∣A∣)) . Let π 1 be a uniform policy.

2: Run (Cheng et al., 2023, Algorithm 1) for T 0 episodes and get outputs ˆ ϕ ∈ Φ , ˆ µ ∈ Υ .

3: Define transitions ̂ P 1 ∶ H -1 as

4: for t = T 0 + 1 , T 0 + 2 , . . . , T do

6: Update policy for all h ∈ [ H ] :

5: Execute policy π t and observe trajectory ( x 1 ∶ H , a 1 ∶ H ) and full information loss ℓ t .

7: end for

As argued in Zhao et al. (2024), the challenge of learning adversarial low-rank MDPs lies in the need for balancing exploration and exploitation both in representation learning and policy optimization over adversarial losses. To tackle this doubled exploration and exploitation challenge, the algorithm of Zhao et al. (2024) performs simultaneous representation learning and policy optimization. With a closer look at their analysis, we find that there is a drawback of this approach: because their algorithm handles the two tasks at the same time, it spends less exploration for representation learning in the early phase of the algorithm. This results in larger error in the estimated Q-values (i.e., critic) fed to policy optimization, and worsens the overall regret.

To address this issue, we design Algorithm 1 as a simple two-phase algorithm that separates representation learning and policy optimization. In the first phase, following Cheng et al. (2023), we perform optimal reward-free exploration for low-rank MDPs to estimate the transition. The resulted estimator, ̂ P , is able to accurately approximate the true transition and give accurate Q-value estimators for any policy. The more accurate Q-estimator allows for more effective policy optimization in the second phase. Theorem 3.1 shows the guarantee of Algorithm 1 where ̃ O hides logarithmic factors of d, H, T, ∣ Φ ∣∣ Υ ∣ .

The proof for Theorem 3.1 is given in Appendix B. Zhao et al. (2024) also constructs a lower bound Ω ( H √ d ∣A∣ T ) for this settings. Thus, the poly (∣A∣) -dependence is unavoidable.

3.2 Model-Based, Computationally Inefficient Algorithm for Bandit Feedback

In this section, following Assumption 3.1, we introduce the first (model-based) algorithm (Algorithm 2) for adversarial low-rank MDPs with bandit feedback and sublinear regret. Compared with linear MDPs, the key challenge for more general low-rank MDPs is to construct a proper loss estimator. For linear MDP, since the feature is known, the loss estimator closely resembles that of linear bandits. However, low-rank MDPs lack such structural simplicity, making standard loss estimators invalid. To overcome this challenge, we propose a new loss estimator that works for any loss function based on off-policy evaluation and the low-rank structure of transition. In this section, ̃ O hides logarithmic factors of d, H, T, ∣ Φ ∣∣ Υ ∣ .

In Algorithm 2, we first conduct an initial representation learning phase to establish accurate transition estimator ̂ P and its corresponding features ˆ ϕ and ˆ µ based on reward-free exploration algorithms in Cheng et al. (2023). Then, in the second phase, we use exponential weights to maintain a distribution over the policy space Π ′ where we mix a uniform policy with Π to enhance exploration. At every round t , a behavior policy π t is chosen from the current policy distribution, and we use the data collected by π t to estimate the value for every π ∈ Π ′ . The success of such off-policy evaluation is based on the following observations of low-rank MDP. Using the low-rank transition structure, for h ≥ 2 , we have

Algorithm 2 Model-Based Algorithm for Bandit Feedback

Input: A policy class Π .

1: Set ϵ = T -1 3 , γ = T -1 3 , β = T -1 3 , η = ( 4 Hd ∣A∣) -1 T -2 3 , and T 0 = ̃ O( ϵ -2 H 3 d 2 ∣A∣( d 2 +∣A∣)) .

3: Define transitions ̂ P 1 ∶ H -1 as

2: Run (Cheng et al., 2023, Algorithm 1) for T 0 episodes and get outputs ˆ ϕ ∈ Φ , ˆ µ ∈ Υ .

4: For all h ∈ [ H -1 ] , define ˆ ϕ h ( π ) = ∑ x,a ˆ d π h ( x, a ) ⋅ ˆ ϕ h ( x, a ) , where ˆ d π h ∶= d ̂ P,π h .

6: for t = T 0 + 1 , T 0 + 2 , . . . , T do

( )∈X×A 5: Define the policy space Π ′ = { π ′ ∶ ∃ π ∈ Π , π ′ h (⋅ ∣ x ) = ( 1 -β ) π h (⋅ ∣ x ) + β /∣A∣ , ∀ x, h } .

7: Define p t ( π ) ∝ exp (-η ∑ t -1 i = 1 ( ˆ ℓ i ( π ) -b i ( π ))) , for all π ∈ Π ′ .

-= 9: Execute policy π t ∼ ρ t and observe trajectory ( x t 1 H , a t 1 H ) and losses ℓ t h = ℓ t h ( x t h , a t h ) . √

8: Let ρ t ( π ) = ( 1 -γ ) p t ( π ) + γ H 1 ∑ H -1 h 1 J h , where J h = John ( ˆ ϕ h (⋅) , Π ′ ) . // John as in §2

∶ ∶ 10: Define Σ t h = ∑ π ∈ Π ′ ρ t ( π ) ⋅ ˆ ϕ h ( π ) ˆ ϕ h ( π ) ⊺ , b t ( π ) = dHϵ ⋅ ∑ H -1 h = 1 ∥ ˆ ϕ h ( π )∥ ( Σ t h ) -1 , and

11: end for

Thus, using the definition of the V -function from Section 2, we have for any loss function ℓ and π :

Letting Λ t h ∶= ( E π t ∼ p t [ ϕ ⋆ h ( π t ) ϕ ⋆ h ( π t ) ⊺ ]) -1 and ingnoring the loss term E [ ℓ 1 ( x 1 , a 1 )] from the first step (this term can easily be treated as in a bandit setting with H = 1 ), we have for all π ∈ Π ′ :

Thus, for all ℓ and π , ∑ H h = 2 ϕ ⋆ h -1 ( π ) ⊺ Λ t h -1 ϕ ⋆ h -1 ( π t ) ⋅ π h ( a h ∣ x h ) π t h ( a h ∣ x h ) -1 ⋅ ℓ h ( x h , a h ) for π t ∼ p t and ( x h , a h ) ∼ d π t h is an unbiased estimator of V π 1 ( x 1 , ℓ ) . However, ϕ ⋆ h -1 ( π ) is not accessible because both the true feature ϕ ⋆ and occupancy d π for the true transition are unknown. Thus, our estimator incorporates the learned feature ˆ ϕ and the occupancy of ̂ P instead as shown in Line 4 and Line 10. Utilizing estimated features and transition could introduce additional bias but the initial representation learning already ensures such bias is small enough to tackle. We compensate for the bias by incorporating an exploration bonus b t ( π ) in exponential weights. To further encourage exploration, we additionally perform John's exploration together with exponential weights when selecting behavior policies. The main guarantee of Algorithm 2 is given in Theorem 3.2.

Note that the guarantee in Theorem 3.2 only holds for policy π ∈ Π . To ensure our regret bound is meaningful, at least a near-optimal policy should be contained in the given policy set Π . In general, the size of such a policy set would grow exponentially with the number of states (e.g. covering of all Markovian policies), making the regret have polynomial dependence on the number of states. In Theorem 3.3, we show that even for low-rank MDPs, if the loss function lacks structure, the regret cannot avoid polynomial dependence on the number of states. The detailed construction for this lower-bound is given in Appendix H.

Theorem 3.3. There exists a low-rank MDP with ∣X∣ states, ∣A∣ actions and sufficiently large T with unstructured losses such that any agent suffers at least regret of Ω ( √ ∣X∣∣A∣ T ) .

Theorem 3.3 shows that under bandit feedback, in general, we could not gain too much from low-rank transition structure compared with tabular MDPs. This contrasts with the Ω ( √ ∣A∣ T ) lower bound in the full information settings (Zhao et al. (2024)). To get rid of any dependence on the number of states, we additionally introduce Assumption 2.2 to impose linear structure on the loss function. Unlike linear MDPs that require the loss feature to be known, our algorithm can even handle linear loss with unknown feature, since our Algorithm 2 never explicitly uses the loss feature. The linear structure is only used to control the size of the candidate policy class in the analysis (i.e., making log ∣ Π ∣ irrelevant to the number of states). Specifically, when both loss and transition are linear, the Q-function is also linear, making it sufficient to consider the following linear policy space:

The 1 T -cover of Π lin only have size ∣ Φ ∣ ⋅ T O( d ) following standard arguments (e.g, Exercise 27.6 of Lattimore and Szepesvári (2020)) and if we feed it into Algorithm 2, our regret could avoid dependece on the size of state space as shown in Corollary 3.1.

Corollary 3.1. If the loss function satisfies Assumption 2.2, applying Algorithm 2 with Π as the 1 T -cover of Π lin ensures Reg T ≤ ̃ O( d 3 H 3 ∣A∣( d 2 +∣A∣) T 2 / 3 ) .

4 Model-free Algorithms for Adversarial Low-rank MDPs

In this section, we consider the model-free setting, where we only assume access to a feature class Φ that contains the true feature map ϕ ⋆ .

Assumption 4.1 (Model-free realizability) . The learner has access to a function class Φ such that

This is a standard assumption in the context of model-free RL (Modi et al., 2024; Zhang et al., 2022b; Mhammedi et al., 2024a). We note that having access to the function class Φ alone (instead of both Φ and Υ as in Assumption 3.1) is not sufficient to model the transition probabilities (unlike in the model-based case). This makes the model-free setting much more challenging; in fact, until the recent work by Mhammedi et al. (2023) there were no model-free, oracle-efficient algorithms for this setting that do not require any additional structural assumptions on the MDP.

4.1 Model-free, Inefficient Algorithm for Bandit Feedback

Our first algorithm follows the same structure as Algorithm 2 but incorporates a model-free initial exploration phase introduced by Mhammedi et al. (2023). Unlike the model-based exploration phase, which directly provides an estimated transition, the model-free exploration phase outputs a policy cover. This policy cover can be combined with the optimization in Algorithm 1 of Liu et al. (2023) to solve the expected feature ˆ ϕ , which is then used in the loss estimator in Line 10 of Algorithm 2. The algorithm, summarized in Algorithm 5, is inefficient but achieves T 2 / 3 regret. More details and proofs can be found in Appendix D.

4.2 Model-free, Oracle Efficient Algorithm for Bandit Feedback (Oblivious Adversary)

We now descibe the key component of our efficient model-free algorithm (Algorithm 3).

Exploration phase and policy cover. Similar to algorithms in the previous section, Algorithm 3 begins with a reward-free exploratorion phase; Line 2 of Algorithm 3. However, unlike in the previous sections where the role of this exploration phase was to learn a model for the transition probabilities, here the goal is to compute a, so called, policy cover which is a small set of policies that can be used to effectively explore the state space.

Definition 4.1 (Approximate policy cover) . For α,ε ∈ ( 0 , 1 ] and h ∈ [ H ] , a subset Ψ ⊆ Π is an ( α,ε ) -policy cover for layer h if

Algorithm 3 Oracle Efficient Algorithm for Adversarial Low-Rank MDPs (Oblivious Adversary).

Input: Number of rounds T , feature class Φ , confidence parameter δ ∈ ( 0 , 1 ) .

1: Set ε ← T -1 / 3 , N reg ← T 2 / 3 , ν ← N -1 / 2 reg , and T 0 ← ε -2 Ad 13 H 6 log ( Φ / δ ) .

cov

∶ 3: for k = 1 , . . . , ( T -T 0 )/ N reg do

2: Get Ψ 1 H ← VoX ( Φ , ε, δ ) . // Compute policy cover with VoX (Mhammedi et al., 2023).

4: Define ̂ π ( k ) h ( a ∣ x ) ∝ exp (-η ∑ s < k ̂ Q ( s ) h ( x, a )) for h ∈ [ H ] .

5: for t = T 0 +( k -1 ) ⋅ N reg + 1 , . . . , T 0 + k ⋅ N reg do

6: Sample variables ζ t ∼ Ber ( ν ) , h t ∼ unif ([ H ]) , and π t ∼ unif ( Ψ cov h t ) .

7: Set ̂ π t = I { ζ t = 0 } ⋅ ̂ π ( k ) + I { ζ t = 1 } ⋅ π t ○ h t π unif ○ h t + 1 ̂ π ( k ) .

8: Execute ̂ π t , and observe trajectory ( x t 1 , a t 1 , . . . , x t H , a t H ) .

9: For h ∈ [ H ] , observe loss ℓ t h ∶= ℓ t h ( x t h , a t h ) .

10: end for

11: For h ∈ [ H ] and I ( k ) = { T 0 +( k -1 ) ⋅ N reg + 1 , . . . , T 0 + k ⋅ N reg } , compute ( ˆ ϕ ( k ) h , ˆ θ ( k ) h ) :

13: end for

In Line 2, Algorithm 3 calls VoX (Mhammedi et al., 2023), a reward-free and model-free exploration algorithm to compute ( 1 /( 8 Ad ) , ε ) -policy covers Ψ cov 1 , . . . , Ψ cov H for layers 1 , . . . , H , respectively, with ∣ Ψ h ∣ = d for all h ∈ [ H ] . This call to VoX requires O ( 1 / ε 2 ) episodes; see the guarantee of VoX in Lemma G.1. After this initial phase, the algorithm operates in epochs, each consisting of N reg ∈ N episodes, where in each epoch k ∈ [ K ] , the algorithm commits to executing policies sampled from a fixed policy distribution ρ ( k ) ∈ ∆ ( Π ) with support on the policy covers Ψ cov 1 ∶ H and a policy ̂ π ( k ) specified by an online learning algorithm. Next, we describe in more detail how ρ ( k ) is constructed and motivate the elements of its construction starting with the online learning policies {̂ π ( k ) } k ∈[ K ] .

Online learning policies. Given estimates { ̂ Q ( s ) 1 ∶ H } s < k of the average Q -functions

from the previous epoch (we will describe how these estimates are computed in the sequel), Algorithm 3 computes policy ̂ π ( k ) for epoch k according to

for all h ∈ [ H ] . Given a state x ∈ X , such exponential weight update ensures a sublinear regret with respect to the sequence of loss functions given by { π (⋅ ∣ x ) ↦ ∑ h ∈[ H ] ̂ Q ( k ) h ( x, π h (⋅ ∣ x ))} k ∈[ K ] . Thanks to the performance difference lemma, and as shown in Luo et al. (2021), a sublinear regret with respect to these "surrogate" loss functions translates into a sublinear regret in the low-rank MDP game we are interested in, granted that { ̂ Q ( k ) 1 ∶ H } k ∈[ K ] are good estimates of the average Q -functions (Luo et al., 2021). In line with previous analyses, we require the Q -function estimates to ensure that the following bias term

is small for all h ∈ [ H ] , k ∈ [ K ] , and π ∈ Π .

Q -function estimates. Thanks to the low-rank MDP structure and the linear loss representation assumption (Assumption 2.2), the avegare Q -functions in (11) are linear in the feature maps ϕ ⋆ . Thus, using the function class Φ in Assumption 2.1 we can estimate these average Q -functions by regressing the sum of losses ∑ H s = h ℓ t s onto ( x t h , a t h ) for t in the k th epoch (as in (9)). However, naïvely doing this using only trajectories generated by ̂ π ( k ) would only ensure that the bias term in (13) is small for π = ̂ π ( k ) . To ensure that it is small for all possible policies π 's, we need to estimate the Q -functions on the trajectories of policies that are guaranteed to have good state coverage; this is where we use the policy cover from the initial phase.

Mixture of policies. At episode t in each epoch k ∈ [ K ] , we execute policy ̂ π t sampled from ρ ( k ) , where ρ ( k ) is the distribution of the random policy:

with ζ t ∼ Ber ( ν ) , h t ∼ unif ([ H ]) , and π t ∼ unif ( Ψ cov h t ) . In words, at the start of each episode of any epoch k , we execute ̂ π ( k ) (see (12)) with probability 1 -ν ; and with probability ν , we execute a policy in Ψ cov 1 ∶ H selected uniformly at random. As explained in the previous paragraph, this ensures a small bias for all choices of π in (13) thanks the policy cover property of Ψ cov 1 ∶ H . We now state the guarantee of Algorithm 3.

Theorem 4.1. Let δ ∈ ( 0 , 1 ) be given and suppose Assumption 2.1 and Assumption 2.2 hold. This, for T = poly ( A,d,H, log (∣ Φ ∣/ δ )) sufficiently large, Algorithm 3 guarantees Reg T ≤ poly ( A,d,H, log (∣ Φ ∣/ δ )) ⋅ T 4 / 5 regret against an oblivious adversary.

The proof is in Appendix E. Note that the T -dependence in this regret even outperforms that of the previous best bound by Zhao et al. (2024) (see Table 1). Compared to their algorithm, Algorithm 3 is model-free and only requires bandit feedback. This makes the result in Theorem 4.1 rather surprising.

4.3 Model-free, Oracle Efficient Algorithm (Adaptive Adversary)

In this section, we present a variant of Algorithm 3 (Algorithm 4) that guarantees a sublinear regret against an adaptive adversary. Given the difficulty of this setting, we make the additional assumption that the algorithm has access to the loss feature ϕ loss , which may be different than the low-rank MDP feature ϕ ⋆ (unlike in Assumption 2.2).

Assumption 4.2 (Loss Representation) . There is a (known) feature map ϕ loss satisfying sup h ∈[ H ] , ( x,a )∈X×A ∥ ϕ loss h ( x, a )∥ ≤ 1 and such that for any round h ∈ [ H ] , t ∈ [ T ] , and history H t -1 = ( x 1 ∶ t -1 1 ∶ H , a 1 ∶ t -1 1 ∶ H ) , the loss function at round t satisfies

for some g t h ∈ B d ( 1 ) .

Note that Assumption 4.2 asserts that the loss at round t depends only on the history H t -1 and the current state action pair. Before moving forward, we introduce some additional notation we will use throughout this section.

Additional notation. For any two feature maps ϕ, ψ ∶ X ×A → R d , we denote by [ ϕ, ψ ] ∶ X × A → R 2 d the vertical concatenation of the two feature maps. For any h ∈ [ H ] , t ∈ [ T ] , policy π ∈ Π , and history H t -1 = ( x 1 ∶ t -1 1 ∶ H , a 1 ∶ t -1 1 ∶ H ) , we denote by Q π h (⋅ , ⋅ ; H t -1 ) the Q -function at layer h corresponding to rollout policy π ; that is,

Finally, we let V π h ( x ; H t -1 ) ∶= max a ∈A Q π h ( x, a ; H t -1 ) denote the corresponding V -function.

Algorithm 4 is similar to Algorithm 3 with the following key differences; after computing a policy cover, the algorithm calls RepLearn (a representation learning algorithm initially introduced by Modi et al. (2024) and subsequently refined by Mhammedi et al. (2023)) to compute a feature map ϕ rep . Then, for every h ∈ [ H ] , the algorithm computes a spanner ; a set of policies Ψ span h = { π h, 1 , . . . , π h, 2 d } that act as an approximate spanner for the set { E π [ ϕ rep h ( x h , a h ) , ϕ loss h ( x h , a h )] ∶ π ∈ Π } ⊆ R 2 d , where we use [⋅ , ⋅] to denote the vertical concatenation of vectors. These spanner

policies are then used as the exploratory policies after the initial phase; that is, at episode t in each epoch k ∈ [ K ] , we execute policy π ( k ) sampled from ρ ( k ) , where ρ ( k ) is set to be the distribution of the random policy: I { ζ t = 0 } ⋅ ̂ π ( k ) + I { ζ t = 1 } ⋅ π t ○ h t + 1 ̂ π ( k ) , with ζ t ∼ Ber ( ν ) , h t ∼ unif ([ H ]) , and π t ∼ unif ( Ψ span h t ) . Here, the main difference to Algorithm 3 (see also (47)) is that we use π t ∼ unif ( Ψ span h t ) instead of π t ∼ unif ( Ψ cov h t ) . We require these spanner policies instead of policies in the policy cover, as an adaptive adversary's history-dependent losses prevent standard least squares regression due to the lack of permutation invariance of state-action pairs across episodes within an epoch. Estimating the Q-functions is thus more complex, and we approach it in expectation over roll-ins using policies in Ψ span , the 'in-expectation' estimation task is in a sense easier.

We now state the guarantee of Algorithm 4.

Theorem 4.2. Let δ ∈ ( 0 , 1 ) be given and suppose that Assumption 2.1 and Assumption 4.2 hold. Then, for T = poly ( A,d,H, log (∣ Φ ∣/ δ )) sufficiently large, Algorithm 4 guarantees with probability at least 1 -δ ,

where π t is the policy that Algorithm 4 executes at episode t ∈ [ T ] .

Algorithm 4 Oracle Efficient Algorithm for Adversarial Low-Rank MDPs (Adaptive Adversary).

Input: Number of rounds T , feature class Φ , loss feature ϕ loss , confidence parameter δ ∈ ( 0 , 1 ) .

2: Set T rep ← α -1 ε -2 AH log (∣ Φ ∣/ δ ) , T span ← α -2 ε -2 A log ( dH ∣ Φ ∣ ε -1 δ -1 ) ,

4: Get Ψ cov 1 H ← VoX ( Φ , ε, δ / 4 ) .

3: Define F h = {( x, a ) ↦ max a ∈A ¯ ϕ h ( x, a ) ⊺ ¯ θ ∣ ¯ ϕ h = [ ϕ loss h , ϕ h ] , ϕ ∈ Φ , ¯ θ ∈ B 2 d ( 1 )} , ∀ h ∈ [ H ] .

∶ 5: Get ϕ rep h ← RepLearn ( h, F h + 1 , Φ , unif ( Ψ cov h ) , T rep ) , for all h ∈ [ H -1 ] . // RepLearn as in Mhammedi et al. (2023)

6: For all h ∈ [ H ] , set ¯ ϕ rep h ←[ ϕ loss h , ϕ rep h ] ∈ R 2 d .

8: Set T 0 ← T cov + T rep + T span .

9:

for

k

=

1

, . . . ,

(

T

-

T

0

)/

N

reg do

12:

Define the random variables

ζ

t

∼

Ber

(

ν

)

,

h

t

∼

unif

t

t

t

t

([

a

H

t

14:

16:

Execute

̂

π

, and observe trajectory end for

5 Conclusion

In this paper, we focus on learning low-rank MDPs with unknown transitions and adversarial losses. For the full-information setting, we improve upon previous regret bounds. More importantly, we initiate the study of the challenging bandit feedback setting, developing various algorithms that achieve sublinear regret under different assumptions. However, the optimal √ T regret remains out of reach due to the limitations of our two-phase design. An interesting direction for future work is to perform on-the-fly representation learning to adapt to adversarial losses and achieve optimal regret.

])

)

.

, . . . ,

x

,

a

,

x

(

, and

π

t

∼

unif

(

Ψ

span

h

t

)

.

References

Abbeel, P. and Ng, A. Y. (2005). Exploration and apprenticeship learning in reinforcement learning. In Proceedings of the 22nd international conference on Machine learning , pages 1-8.

Agarwal, A., Kakade, S., Krishnamurthy, A., and Sun, W. (2020). Flambe: Structural complexity and representation learning of low rank mdps. Advances in neural information processing systems , 33:20095-20107.

Ayoub, A., Jia, Z., Szepesvari, C., Wang, M., and Yang, L. (2020). Model-based reinforcement learning with value-targeted regression. In International Conference on Machine Learning , pages 463-474. PMLR.

Cai, Q., Yang, Z., Jin, C., and Wang, Z. (2020). Provably efficient exploration in policy optimization. In International Conference on Machine Learning , pages 1283-1294. PMLR.

Chen, F., Mei, S., and Bai, Y . (2022a). Unified algorithms for rl with decision-estimation coefficients: No-regret, pac, and reward-free learning. arXiv preprint arXiv:2209.11745 .

Chen, J., Modi, A., Krishnamurthy, A., Jiang, N., and Agarwal, A. (2022b). On the statistical efficiency of reward-free exploration in non-linear rl. Advances in Neural Information Processing Systems , 35:20960-20973.

Chen, L. and Luo, H. (2021). Finding the stochastic shortest path with low regret: The adversarial cost and unknown transition case. In International Conference on Machine Learning , pages 1651-1660. PMLR.

Cheng, Y., Huang, R., Yang, J., and Liang, Y. (2023). Improved sample complexity for reward-free reinforcement learning under low-rank mdps. arXiv preprint arXiv:2303.10859 .

Dai, Y., Luo, H., and Chen, L. (2022). Follow-the-perturbed-leader for adversarial markov decision processes with bandit feedback. Advances in Neural Information Processing Systems , 35:1143711449.

Dai, Y., Luo, H., Wei, C.-Y., and Zimmert, J. (2023). Refined regret for adversarial mdps with linear function approximation. In International Conference on Machine Learning , pages 6726-6759. PMLR.

Dann, C., Wei, C.-Y., and Zimmert, J. (2023). Best of both worlds policy optimization. In International Conference on Machine Learning .

Du, S., Kakade, S., Lee, J., Lovett, S., Mahajan, G., Sun, W., and Wang, R. (2021). Bilinear classes: A structural framework for provable generalization in rl. In International Conference on Machine Learning , pages 2826-2836. PMLR.

Foster, D. J., Kakade, S. M., Qian, J., and Rakhlin, A. (2021). The statistical complexity of interactive decision making. arXiv preprint arXiv:2112.13487 .

Foster, D. J., Rakhlin, A., Sekhari, A., and Sridharan, K. (2022). On the complexity of adversarial decision making. Advances in Neural Information Processing Systems , 35:35404-35417.

He, J., Zhou, D., and Gu, Q. (2022). Near-optimal policy optimization algorithms for learning adversarial linear mixture mdps. In International Conference on Artificial Intelligence and Statistics , pages 4259-4280. PMLR.

Huang, A., Chen, J., and Jiang, N. (2023). Reinforcement learning in low-rank mdps with density features. In International Conference on Machine Learning , pages 13710-13752. PMLR.

Jiang, N., Krishnamurthy, A., Agarwal, A., Langford, J., and Schapire, R. E. (2017). Contextual decision processes with low bellman rank are pac-learnable. In International Conference on Machine Learning , pages 1704-1713. PMLR.

Jin, C., Jin, T., Luo, H., Sra, S., and Yu, T. (2020a). Learning adversarial markov decision processes with bandit feedback and unknown transition. In International Conference on Machine Learning , pages 4860-4869. PMLR.

Jin, C., Liu, Q., and Miryoosefi, S. (2021a). Bellman eluder dimension: New rich classes of rl problems, and sample-efficient algorithms. Advances in neural information processing systems , 34:13406-13418.

Jin, C., Yang, Z., Wang, Z., and Jordan, M. I. (2020b). Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory , pages 2137-2143. PMLR.

Jin, T., Huang, L., and Luo, H. (2021b). The best of both worlds: stochastic and adversarial episodic mdps with unknown transition. Advances in Neural Information Processing Systems , 34:20491-20502.

Kakade, S. and Langford, J. (2002). Approximately optimal approximate reinforcement learning. In Proceedings of the Nineteenth International Conference on Machine Learning , pages 267-274.

Kong, F., Zhang, X., Wang, B., and Li, S. (2023). Improved regret bounds for linear adversarial mdps via linear optimization. arXiv preprint arXiv:2302.06834 .

Lattimore, T. and Szepesvári, C. (2020). Bandit algorithms . Cambridge University Press.

Lee, C.-W., Luo, H., Wei, C.-Y., and Zhang, M. (2020). Bias no more: high-probability datadependent regret bounds for adversarial bandits and mdps. In Advances in Neural Information Processing Systems .

Liu, H., Wei, C.-Y., and Zimmert, J. (2023). Towards optimal regret in adversarial linear mdps with bandit feedback. arXiv preprint arXiv:2310.11550 .

Luo, H., Wei, C.-Y., and Lee, C.-W. (2021). Policy optimization in adversarial mdps: Improved exploration via dilated bonuses. Advances in Neural Information Processing Systems , 34:2293122942.

Mhammedi, Z., Block, A., Foster, D. J., and Rakhlin, A. (2023). Efficient model-free exploration in low-rank mdps. arXiv preprint arXiv:2307.03997 .

Mhammedi, Z., Block, A., Foster, D. J., and Rakhlin, A. (2024a). Efficient model-free exploration in low-rank mdps. Advances in Neural Information Processing Systems , 36.

Mhammedi, Z., Foster, D. J., and Rakhlin, A. (2024b). The power of resets in online reinforcement learning. arXiv preprint arXiv:2404.15417 .

Modi, A., Chen, J., Krishnamurthy, A., Jiang, N., and Agarwal, A. (2024). Model-free representation learning and exploration in low-rank mdps. Journal of Machine Learning Research , 25(6):1-76.

Padakandla, S., KJ, P., and Bhatnagar, S. (2020). Reinforcement learning algorithm for non-stationary environments. Applied Intelligence , 50(11):3590-3606.

Ren, T., Zhang, T., Szepesvári, C., and Dai, B. (2022). A free lunch from the noise: Provable and practical exploration for representation learning. In Uncertainty in Artificial Intelligence , pages 1686-1696. PMLR.

Rosenberg, A. and Mansour, Y. (2019). Online stochastic shortest path with bandit feedback and unknown transition function. Advances in Neural Information Processing Systems , 32.

Shani, L., Efroni, Y., Rosenberg, A., and Mannor, S. (2020). Optimistic policy optimization with bandit feedback. In International Conference on Machine Learning , pages 8604-8613. PMLR.

Sherman, U., Cohen, A., Koren, T., and Mansour, Y. (2023a). Rate-optimal policy optimization for linear markov decision processes. arXiv preprint arXiv:2308.14642 .

Sherman, U., Koren, T., and Mansour, Y. (2023b). Improved regret for efficient online reinforcement learning with linear function approximation. In International Conference on Machine Learning .

Uehara, M., Zhang, X., and Sun, W. (2021). Representation learning for online and offline rl in low-rank mdps. arXiv preprint arXiv:2110.04652 .

Xie, T., Foster, D. J., Bai, Y ., Jiang, N., and Kakade, S. M. (2022). The role of coverage in online reinforcement learning. arXiv preprint arXiv:2210.04157 .

Zhang, T., Ren, T., Yang, M., Gonzalez, J., Schuurmans, D., and Dai, B. (2022a). Making linear mdps practical via contrastive representation learning. In International Conference on Machine Learning , pages 26447-26466. PMLR.

Zhang, X., Song, Y., Uehara, M., Wang, M., Agarwal, A., and Sun, W. (2022b). Efficient reinforcement learning in block mdps: A model-free representation learning approach. In International Conference on Machine Learning , pages 26517-26547. PMLR.

Zhao, C., Yang, R., Wang, B., and Li, S. (2022). Learning adversarial linear mixture markov decision processes with bandit feedback and unknown transition. In The Eleventh International Conference on Learning Representations .

Zhao, C., Yang, R., Wang, B., Zhang, X., and Li, S. (2024). Learning adversarial low-rank markov decision processes with unknown transition and full-information feedback. Advances in Neural Information Processing Systems , 36.

Zhong, H., Xiong, W., Zheng, S., Wang, L., Wang, Z., Yang, Z., and Zhang, T. (2022). Gec: A unified framework for interactive decision making in mdp, pomdp, and beyond. arXiv preprint arXiv:2211.01962 .

Appendices

A Related Work

Learning low-rank MDPs in the stochastic setting. In the absent of adversarial losses, several general learning frameworks have been developed for super classes of low-rank MDPs which offer tight sample complexity for either reward-based (Jiang et al., 2017; Jin et al., 2021a; Du et al., 2021; Foster et al., 2021; Zhong et al., 2022) or reward-free (Chen et al., 2022b,a; Xie et al., 2022) settings. However, these algorithms require solving non-convex optimization problems on non-convex version spaces, making them computationally inefficiency. Oracle-efficient algorithms for low-rank MDPs are first obtained by Agarwal et al. (2020) using a model-based approach, and the sample complexity bound has been largely improved in subsequent works (Uehara et al., 2021; Zhang et al., 2022a; Cheng et al., 2023). The model-based approach, however, necessitates the function class to accurately model the transition, which is a strong requirement. To relax it, Modi et al. (2024); Zhang et al. (2022b) developed oracle-efficient model-free algorithms, but both of them require additional assumptions on the MDP structure. Recently, Mhammedi et al. (2024a) proposed a satisfactory model-free algorithm that removes all these assumptions. Our work leverages their techniques to tackle the more challenging adversarial setting.

Learning adversarial MDPs. Learning adversarial tabular MDPs under bandit feedback and unknown transition has been extensively studied (Rosenberg and Mansour, 2019; Jin et al., 2020a; Lee et al., 2020; Jin et al., 2021b; Shani et al., 2020; Chen and Luo, 2021; Luo et al., 2021; Dai et al., 2022; Dann et al., 2023). This line of work has demonstrated not only √ T regret bounds but also several data-dependent bounds.

For adversarial MDPs with a large state space which necessitates the use of function approximation, if the transition is known, Foster et al. (2022) shows that adversarial setting is as easy as the stochastic setting even under general function approximation. For full-information loss feedback with unknown transition, √ T bound is derived for both linear mixture MDPs (Cai et al., 2020; He et al., 2022) and linear MDPs (Sherman et al., 2023a). For more challenging low-rank MDPs with unknown features, the best result only achieves T 5 / 6 regret (Zhao et al., 2024).

For function approximation with bandit feedback and unknown transition, Zhao et al. (2022) provides √ T bound for linear mixture MDPs, but their regret has polynomial dependence on the size of the state space due to the lack of structure on the loss function. For linear MDPs, a series of recent work has made significant progress in improving the regret bound (Luo et al., 2021; Dai et al., 2023; Sherman et al., 2023b; Kong et al., 2023; Liu et al., 2023). The state-of-the-art result by Liu et al. (2023) gives an inefficient algorithm with √ T regret and an efficient algorithm with T 3 / 4 regret. These regret bounds for linear MDPs do not depend on the state space size because of the linear loss assumption. We show in Appendix H that cross-state structure on the losses is necessary for low-rank MDPs with bandit feedback to achieve regret bound that do not scale with the number of states.

B Proof of Theorem 3.1 (Model-Based, Full Information)

Theorem B.1 (Theorem 3 in Cheng et al. (2023)) . With probability 1 -δ , for any policy π and layer h , Algorithm 1 in Cheng et al. (2023) outputs transition ̂ P 1 ∶ H and features ˆ ϕ h , ˆ µ h such that ̂ P h ( x ′ ∣ x, a ) = ˆ ϕ h ( x, a ) ⊺ ˆ µ h + 1 ( x ′ ) and

if the number of collected trajectories is at least O( H 3 d 2 ∣A∣( d 2 +∣A∣) ϵ 2 log 2 ( TdH ∣ Φ ∣∣ Υ ∣)) .

Define ̂ V π 1 ( x 1 ; ℓ ) as the value function of policy π under transition { ̂ P h } H h = 1 and loss ℓ . We have

Bounding the bias term. By Theorem B.1 and Lemma I.6, we have

Bounding the FTRL term. Since η = 1 H √ T , we have

Thus, by setting ϵ = ( Hd 2 ∣A∣( d 2 +∣A∣)) 1 3 T -1 3 , we have

C Proof of Theorem 3.2 (Model-Based, Bandit Feedback)

Lemma C.1. With ̂ P 1 ∶ H as in Theorem B.1, we have for any h ∈ [ H ] and any policy π ,

where ˆ d π h ∶= d ̂ P,π h

Proof. We prove the claim by induction. When h = 1 , given that the ∥ d π 1 -ˆ d π 1 ∥ 1 = 0 . Assume

We have

where the last step follows by the induction hypothesis. The second inequality of Lemma C.1 directly comes from Theorem B.1.

Our candidate policy space Π ′ has a β mixture of the random policy. For any deterministic policy π ⋆ 0 , define policy π ⋆ such that for any state x ∈ X , we have π ⋆ (⋅ ∣ x ) = ( 1 -β ) π ⋆ 0 (⋅ ∣ x ) + β ∣A∣ . We have π ⋆ ∈ Π ′ . Define ̂ V π 1 ( x 1 ; ℓ ) as the value function of policy π under transition { ̂ P h } H h = 1 and loss ℓ .

For any policy π ⋆ 0 , we have

Recall that J = John { ˆ ϕ h ( π )} π ∈ Π ′ ,h ∈[ H ] ∈ ∆ ( Π ′ × H ) , and we have ρ t ( π ) = ( 1 -γ ) p t ( π ) + γ ∑ H h = 1 J ( π, h ) where p t ( π ) is defined in Line 7 of Algorithm 2.

Proof.

where the last step follows by Lemma I.7.

Lemma C.3. We have

Proof. This is a direct result combing Theorem B.1 and Lemma I.6.

For ( x 1 ∶ H , a 1 ∶ H ) ∈ X H ×A H and π ∈ Π ′ where Π ′ defined in Line 5 in Algorithm 2 is the mix of a given policy class Π and a uniform policy, and is also the policy class we play with. Recall that the loss estimator

defined in Line 10 of Algorithm 2.

Lemma C.4. For any episode t ∈ [ T ] , for any policy π we have

Proof. First, from the definition in Line 4 of Algorithm 2, for any x ∈ X with h ≥ 2 , we have

where ˆ ϕ h -1 ( π ) = ∑ x ′ ,a ′ ˆ d π h -1 ( x ′ , a ′ ) ˆ ϕ h -1 ( x ′ , a ′ ) .

We now prove the first inequality:

Through importance sampling, we have

First

=

E

π

t

∼

π

π

1

t

1

(

(

a

a

1

1

∣

∣

x

x

1

1

)

)

We now bound the remaining term in (42) Since Σ t h -1 = E π t ∼ ρ t [ ˆ ϕ h -1 ( π t ) ˆ ϕ h -1 ( π t ) ⊺ ] , we have Remain

where (25) follows by Cauchy-Schwarz, and the last inequality uses Lemma C.1. Adding up First and Remain , and using the defintion of ˆ ℓ in (22) implies the first inequality of the lemma.

The second inequality follows the same procedure except for applying Cauchy-Schwarz inequality in the opposite direction in Eq. (24).

Proof. Recall in Line 10 of Algorithm 2, b t ( π ) ∶= √ dHϵ ∑ H h = 2 ∥ ˆ ϕ h -1 ( π )∥ ( Σ t h -1 ) -1 , for all π ∈ Π ′ . By Lemma C.4, we have

Lemma C.5. If η ≤ ( 2 ∣A∣ Hd βγ + dH 2 ϵ √ γ ) -1 , then 2 where EXP is as in (21) .

ρ

t

E

π

t

[

⋅

ℓ

t

1

(

x

1

,

a

1

)]

.

where

We now bound the FTRL term. Since ρ t ( π ) = ( 1 -γ ) p t ( π ) + γ ∑ H h = 1 J ( π, h ) , define Σ J = ∑ π ∈ Π ∑ H h = 1 J ( π, h ) ˆ ϕ h ( π ) ˆ ϕ h ( π ) ⊺ . Note that for any t ∈ [ T ] and h ∈ [ H ] , we have Σ t h ⪰ γ Σ J . By the triangle inequality, we have for any π, π t and ( x 1 ∶ H , a 1 ∶ H ) ,

and

To ensure η ∣ ˆ ℓ t ( π ; π t , x 1 ∶ H , a 1 ∶ H ) -b t ( π )∣ ≤ 1 , it suffices to set η ≤ ( 2 ∣A∣ Hd βγ + dH 2 ϵ √ γ ) -1 . Under this constraint, from Lemma I.5, we have

For any t ∈ [ T ] , we have

where (29) follows by the fact that p t ( π ) ≤ 1 1 -γ ρ t ( π ) and 1 1 -γ ≤ 2 . Thus,

Moreover,

Combing Lemma C.2, Lemma C.3 and Lemma C.5, if η ≤ ( 2 ∣A∣ Hd βγ + dH 2 ϵ √ γ ) -1 , we have

D More Details of Inefficient Model-Free Algorithm in Section 4.1

D.1 Algorithm Description

In this section, we give a more detailed introduction of the algorithm mentioned in Section 4.1. This algorithm is model-free and achieves T 2 3 regret, but it is computationally inefficient. We consider the low-rank MDPs with linear losses that satisfies Assumption 2.1 and Assumption 2.2.

Let C ( S, ϵ ′ ) be ϵ ′ -net of space S . We define necessary policy and function classes in Definition D.1.

Definition D.1. We define linear policy class and its discretization as

Define corresponding function class as follows

Our main algorithm is given in Algorithm 5, which shares the same structure as Algorithm 2, but with a different initial phase to learn expected feature estimator ˆ ϕ h ( π ) = ∑ ( x,a )∈X×A ˆ d π h ( x ) π ( a ∣ x ) ˆ ϕ h ( x, a ) to approximate ϕ ⋆ h ( π ) = ∑ ( x,a )∈X×A d π h ( x ) π ( a ∣ x ) ϕ ⋆ h ( x, a ) for every h ∈ [ H ] . In Algorithm 2, under Assumption 3.1, it is feasible to use established model-based approach to learn an accurate estimated transition ̂ P together with its feature ˆ ϕ . The occupancy estimator ˆ d π 1 ∶ H is induced by ̂ P which also enjoy small errors. However, when we move to model-free settings with Assumption 4.1, there is no existing approach that could guarantee a good estimation for ˆ d π h ( x ) and ˆ ϕ .

To tackle this challenge, we first call VoX (Mhammedi et al., 2023) to construct a policy cover Ψ cov 1 ∶ H , and then play every policy in Ψ cov 1 ∶ H for n episodes to collect data. Subsequently, these data are fed into Algorithm 6 to jointly solve estimated occupancy ˆ d π 1 ∶ H and feature ˆ ϕ 1 ∶ H . Algorithm 6 is similar to (Liu et al., 2023, Algorithm 1), which is used to estimate occupancy on the fly for linear MDP. In Algorithm 6, given a target policy π , we jointly solve ˆ ϕ 1 ∶ H ∈ Φ , ˆ d π 1 ∶ H ∈ [ 0 , 1 ] ∣X∣ , and ( ˆ ξ 1 ∶ H,f ) f ∈F π ⊂ B d ( √ d ) that satisfies four constrains, where ˆ ξ h,f is the estimation of ξ ⋆ h,f ∶= ∑ x ′ ∈X µ ⋆ h + 1 ( x ′ ) f ( x ′ ) . The first constraint Eq. (30) ensures the estimated occupancy ˆ d π 1 ∶ H are valid distrbutions. The second constrant Eq. (31) enforces the estimated values to follow the dynamic programming relationship between the occupancy of layer h and layer h + 1 , which helps to control the propagation of estimation errors across layers through the bias of ˆ ϕ h ( x, a ) ⊺ ˆ ξ h,f . The third constrain Eq. (32) and fourth constrain Eq. (33) are then used to bound the estimated bias of ˆ ϕ h ( x, a ) ⊺ ˆ ξ h,f by utilizing the data collected from policies in policy covers Ψ cov 1 ∶ H . Note that applying Eq. (33) requires access to the whole state space, which is an additional assumption not needed in previous algorithms. The gurantee of Algorithm 5 is given in Theorem D.1, where the ̃ O hides the logarithmic dependence on d, H, ∣A∣ , T .

Algorithm 5 Model-Free Algorithm for Bandit Feedback

Input: Policy class Π = Π cov lin ( 1 T ) .

2: Get Ψ cov 1 H ← VoX ( Φ , ε, δ ) using T 0 episodes.

1: Set ϵ = 18 -1 d 5 2 T -1 3 , γ = T -1 3 , β = T -1 3 , η = ( 4 Hd ∣A∣) -1 T -2 3 , n = 11250 d 5 2 ∣A∣ T 2 3 log 3 dnHT ∣ Φ ∣ δ and T 0 = ̃ O ( ϵ -2 ∣A∣ d 13 H 6 log (∣ Φ ∣/ δ ))) .

∶ 3: For every policy π ′ ∈ Ψ cov 1 ∶ H , play it for n episodes and get the data set (D π ′ h ) h ∈[ H ] where D π ′ h π ′

consists of tuples ( x, a, x ′ ) such that ( x, a ) ∼ d h and x ′ ∼ P ⋆ (⋅ ∣ x, a ) . 4: Define the policy space Π ′ = { π ′ ∶ ∃ π ∈ Π , π ′ h (⋅ ∣ x ) = ( 1 -β ) π h (⋅ ∣ x ) + β /∣A∣ , ∀ x, h } .

5: Get ˆ ϕ h (⋅) ← EOM-PC ( Π ′ , (D π ′ h ) h ∈[ H ] ,π ′ ∈ Ψ cov 1 ∶ H ) from Algorithm 6.

7: Define p t ( π ) ∝ exp (-η ∑ t -1 i = 1 ( ˆ ℓ i ( π ) -b i ( π ))) , for all π ∈ Π ′ .

6: for t = T 0 + 1 , T 0 + 2 , . . . , T do

-9: Execute policy π t ∼ ρ t and observe trajectory ( x t 1 H , a t 1 H ) and losses ℓ t h = ℓ t h ( x t h , a t h ) .

11: end for

Algorithm 6 EOM-PC ( Π , (D π ′ h ) h ∈[ H ] ,π ′ ∈ Ψ cov 1 ∶ H ) (Estimate Occupancy Measure with Policy Cover)

Input: The policy class Π , datasets (D π h ) h ∈[ H ] for every π ′ ∈ Ψ cov 1 ∶ H Jointly find ˆ ϕ h ∈ Φ , ( ˆ d π h ) π ∈ Π ∈ [ 0 , 1 ] ∣X∣ , and ( ˆ ξ h,f ) f ∈F ⊂ B d ( √ d ) for any h ∈ [ H ] such that for all π ∈ Π ,

Output: ˆ ϕ h ∶ Π → R d , ˆ ϕ h ( π ) = ∑ ( x,a )∈X×A ˆ d π h ( x ) π ( a ∣ x ) ˆ ϕ h ( x, a ) , ∀ h ∈ [ H ] .

D.2 Analysis of Occupancy Estimation from Algorithm 6

Lemma D.1. With probability 1 -δ , ϕ ⋆ 1 ∶ H , ( d π 1 ∶ H ) π ∈ Π ′ , and ξ ⋆ h,f ∶= ∑ x ′ ∈X µ ⋆ h + 1 ( x ′ ) f ( x ′ ) , ∀ f ∈ F , ∀ h ∈ [ H ] is a solution to Algorithm 6.

′

Proof. Since for any policy π and any h ∈ [ H ] , ∑ x ∈X d π h ( x ) = 1 , Eq. (30) holds. For any policy π , any f ∈ F π and any h ∈ [ H ] , we have

Thus, Eq. (31) holds. From Exercise 27.6 of Lattimore and Szepesvári (2020), the ϵ -net of B d ( R ) is ( 3 R ϵ ) d . Thus. ∣ Π ′ ∣ = ∣ Π cov lin ( 1 T )∣ = ∣ Φ ∣ ( 3 √ dHT 2 ) d . We also have ∣C ( B d ( √ d ) , 1 T )∣ = ( 3 √ dT ) d and for any policy π , ∣C (F π , 1 T )∣ = ∣ Φ ∣ ( 3 √ dT ) d . To consider all possible instances, define

Thus, by union bound, with probability of 1 -δ , for every π ∈ Π ′ , every π ′ ∈ Ψ cov 1 ∶ H , every f ∈ C (F π , ϵ ) , every ξ ∈ C ( B d ( √ d ) , ϵ ) , every ϕ ∈ Φ and every h ∈ [ H ] , we have

Bounding the distance through 1 T -net, we have with probability of 1 -δ , for every π ∈ Π ′ , every π ′ ∈ Ψ cov 1 ∶ H , every f ∈ F π , every ξ ∈ B d ( √ d ) , every ϕ ∈ Φ and every h ∈ [ H ] ,

Thus, Eq. (32) also holds. Finally, for all x, a , we have

Thus, ϕ ⋆ h , d π h , and ξ ⋆ h,f ∶= ∑ x ′ ∈X µ ⋆ h + 1 ( x ′ ) f ( x ′ ) , ∀ f ∈ F π , h ∈ [ H ] satisfy all Eq. (30) - Eq. (33) and is a solution to Algorithm 6.

Lemma D.2. With probability 1 -δ , for all f ∈ F , any solution ˆ d π from Algorithm 6 for any π ∈ Π ′ satisfies

Proof.

For every solution ˆ ϕ 1 ∶ H , ˆ ξ 1 ∶ H,f ∈F π , ( ˆ d π 1 ∶ H ) π ∈ Π ′ of Algorithm 6, we have

where the last step comes from the constrain Eq. (32). On the other hand

Combing Eq. (34) and Eq. (35), we have

Since for every data tuple ( x, a, x ′ ) ∈ D π ′ h , ( x, a ) ∼ d π ′ h independently, by Lemma I.3, for every π ′ ∈ Ψ cov 1 ∶ H we have

This implies there exists a representation ˆ ϕ 1 ∶ H ( x, a ) such that for every f ∈ F and any h ∈ [ H ] , there exists ˆ ξ h,f such that

Eq. (36) matches Eq. (99) with a different error bound on the right hand. From Lemma G.1, we have Ψ cov h is a ( 1 8 Ad , ε ) -policy cover for layer h , following the rest of the proof in Lemma G.2, for every π and every f ∈ F , h ∈ [ H ] , we have

The d 5 2 in the second term of Eq. (37) improves the d 7 2 term in Eq. (98) because ˆ ξ h,f ∈ B d ( √ d ) rather than B d ( d 3 2 ) . Putting the choice n = 11250 d 5 2 ∣A∣ T 2 3 log 3 dnHT ∣ Φ ∣ δ and ϵ = d 5 2 18 T -1 3 into Eq. (37), we have for every π and every f ∈ F , h ∈ [ H ] , we have

Utilizing above results, for every π , every f ∈ F and any h ∈ [ H -1 ] , we have

where in the last step, we define f ′ ( x ) = ∑ a π h ( a ∣ x ) ˆ ϕ h ( x, a ) ⊺ ˆ ξ h,f ∈ F . This allow us to use recursion to finish the proof.

D.3 Regret Analysis

We begin by proving Lemma D.3, showing that the policy class Π cov lin ( ϵ ′ ) suffices to approximate all policies with small error.

Lemma D.3. For any policy π , there exists a policy π ′ ∈ Π cov lin ( ϵ ′ ) such that

Proof. Let θ t,π h ∈ B d ( H √ d ) be such that

Such a θ π,t h is guaranteed to exist by the low-rank MDP structure (Assumption 2.1) and Assumption 2.2. For every h ∈ [ H ] , since ∑ T t = 1 θ π,t h ∈ B d ( √ dHT ) , we define θ ′ h ∈ C ( B d ( √ dHT ) , ϵ ′ ) such that ∥ θ ′ h - ∑ T t = 1 θ π,t h ∥ 2 ≤ ϵ ′ , and let π h ( a ∣ x ) = I { a = argmin a ∈A ϕ ⋆ h ( x, a ) ⊺ θ ′ h } for every h ∈ [ H ] . We have π ′ ∈ Π cov lin ( ϵ ′ ) . From Lemma I.7, we have

where the last inequality comes from the fact that π h ( a ∣ x ) = I { a = argmin a ∈A ϕ ⋆ h ( x, a ) ⊺ θ ′ h } and ∥ θ ′ h - ∑ T t = 1 θ π,t h ∥ 2 ≤ ϵ ′ for every h ∈ [ H ] .

From Lemma D.3, for any policy π ⋆ 1 , there exists a policy π ⋆ 0 ∈ Π cov lin ( 1 T ) such that

Our candidate policy space Π ′ has a β mixture of the random policy. For any policy π ⋆ 0 ∈ Π cov lin ( 1 T ) , define policy π ⋆ such that for any state x ∈ X , we have π ⋆ (⋅ ∣ x ) = ( 1 -β ) π ⋆ 0 (⋅ ∣ x ) + β ∣A∣ . We have π ⋆ ∈ Π ′ . Define

Utilizing Eq. (39) and the decomposition in Eq. (21), together with the fact that ∣ Ψ cov h ∣ ≤ d from Lemma G.1, we have for any policy π ⋆ 1 ,

Following Lemma C.2, we have

Lemma D.4.

Proof. For any policy π and any t ∈ [ T ] ,

Thus,

We now prove a modle-free counterpart of Lemma C.4 in Lemma D.5.

Lemma D.5. For any episode t ∈ [ T ] , for any policy π we have

Proof.

We now prove the first inequality:

Through importance sampling, we have

We now bound the remaining term in (42) Since Σ t h -1 = E π t ∼ ρ t [ ˆ ϕ h -1 ( π t ) ˆ ϕ h -1 ( π t ) ⊺ ] , we have Remain

Adding up First and Remain , and using the defintion of ˆ ℓ in (22) implies the first inequality of the lemma.

The second inequality follows the same procedure except for applying Cauchy-Schwarz inequality in the opposite direction in Eq. (43).

Given Lemma D.5, we could follow the same procedure in Lemma C.5 except replacing the factor of bonus b t ( π ) from √ dHϵ to d 11 2 HT -1 3 . This leads to the following changes. Firstly, by a similar argument as Eq. (27), we have

where

Secondly, now we have

To ensure η ∣ ˆ ℓ t ( π ; π t , x 1 ∶ H , a 1 ∶ H ) -b t ( π )∣ ≤ 1 , it suffices to set η ≤ ( 2 ∣A∣ Hd βγ + d 6 H 2 T -1 6 ) -1 = ( 2 ∣A∣ HdT 2 3 + d 6 H 2 T -1 6 ) -1 from β = γ = T -1 3 . Thus our choice η = ( 4 Hd ∣A∣) -1 T -2 3 satisfies the condition if we assume T ≥ d 6 H 2 . Moreover, now we have

Putting these two changes back to the proof into Lemma C.5, given η = ( 4 Hd ∣A∣) -1 T -2 3 we have

Putting Eq. (41), Lemma D.4, Eq. (44) into Eq. (40) together with ϵ 18 1 d 2 T 3 , we have

= -5 -1 Reg T ≤ ̃ O ( d 8 H 6 ∣A∣ T 2 3 log (∣ Φ ∣)))

E Proof of Theorem 4.1 (Model-Free, Banfit Feedback)

We start by introducting some notation. We let I ( k ) denote the rounds in the k -th epoch:

where T 0 is as in Line 1 of Algorithm 3. Throughout the analysis, we condition on the event

where E cov is as in Lemma G.1. Further, for any k ∈ [ K ] , let ρ ( k ) be the distribution of the random policy:

We start our analysis by applying the performance difference lemma.

Applying the performance difference lemma. For any k ∈ [ K ] , t ∈ I ( k ) , and ρ ( k ) as just defined, we have

Thus, by the triangle inequality, we have

We start by bound the first term in the right-hand side of (48) (the regret term).

E.1 Bounding the Regret Term

Fix h ∈ [ H ] and define ˜ π ⋆ h (⋅ ∣ x ) ∶= ( 1 -γ / T ) ⋅ π ⋆ h (⋅ ∣ x ) + γπ unif (⋅ ∣ x )/ T for all x , where γ is as in Algorithm 3. We have that ∣ ̂ Q ( k ) h ( x, a )∣ = ∣ ˆ ϕ ( k ) h ( x, a ) ⊺ ˆ θ ( k ) h ∣ ≤ H √ d (since ˆ θ ( k ) h ∈ B d ( H √ d ) and ∥ ˆ ϕ k h ( x, a )∥ ≤ 1 ). By applying Lemma I.5, we have that for any η ≤ 1 H √ d and x ∈ X :

E.2 Bounding the Bias Term

Fix epoch k ∈ [ K ] , round t ∈ I ( k ) , layer h ∈ [ 2 .. H ] , and π ′ ∈ Π . Further, let

be the set of ε -reachable states. With this notation, we now bound the bias term in (48); we have

Thus, by letting

and using Jensen's inequality (twice), we get

and now by the fact that Ψ cov h is an ( 1 8 Ad , ε ) policy cover for layer h (see Lemma G.1 and Definition 4.1):

Next, we bound the regression error term in the right-hand side of (52).

E.3 Bounding the Regression Error

Lemma E.1. Let δ ∈ ( 0 , 1 ) and ν ∈ ( 0 , 1 / 4 ) be given. There is an event E reg of probability at least 1 -2 δ under which

Proof. Fix δ ∈ ( 0 , 1 ) , h ∈ [ H ] and k ∈ [ K ] , and let ( x t h , a t h , ζ t , h t ) be as in Algorithm 3. With this, define

and note that ( x t h , a t h , I t h ) t ∈I ( k ) are identically and independently distributed. Further, for t ∈ I ( k ) and π ∈ Π , let θ t,π h ∈ B d ( H √ d ) be such that

Such a θ π,t h is guaranteed to exist by the low-rank MDP structure (Assumption 2.1) and Assumption 2.2. With this, note that for Q ̂ π ( k ) h as in (51), we have

For the rest of this proof, we let F be the function class

where C is a minimal ( N reg ) -1 -cover of B ( H √ d ) in ∥ ⋅ ∥ distance. Further, for τ ∈ I ( k ) we let and

for f ∈ F . Finally, let f ⋆ ( x, a ) ∶= ϕ ⋆ h ( x, a ) ⊺ θ ( k ) h , where θ ( k ) h is as in (55) and ˆ f ( x, a ) ∶= ̂ Q ( k ) h ( x, a ) . With this, note that f ⋆ and ˆ f satisfy f ⋆ ( x t h , a t h ) = z t h -ε t h and ˆ f ∈ argmin f ∈F ̂ L ( f ) .

Now, since ˆ f ∈ argmin f ̂ L ( f ) , we have

where ∇ denotes directional derivative and

Rearranging (58) and using that f ⋆ ( x t h , a t h ) = z t h -ε t h , we get that

We now bound the right-hand side of (59). For any h ∈ [ H ] , k ∈ [ K ] and any ˆ f, f ⋆ ∈ F , we apply Lemma I.2 with

The random variable w i set as the difference

n = N reg = ∣I ( k ) ∣ .

R = 4 H 2 ; and

λ = 1 /( 16 H 2 ) ;

to get that there is an event E of porbability at least 1 -δ under which

where E t [⋅] is defined as E [⋅ ∣ F t -1 ] and the last step uses that ∣ ε t h ∣ ≤ 2 H , for all t ∈ I ( k ) . For the rest of the proof, we condition on E and to simplify notation let

Using the expression of ε t h in (56), we have

Now, since I t h = I { ζ t = 0 or h t ≤ h } , we have

Plugging this into (62) and using the law of total expectation, we have

On the other hand, since ( x t h , a t h , I t h ) t ∈I ( k ) are i.i.d. and ℓ t is chosen by an oblivious adversary, we have

Plugging this into (63) shows that

Combining this with (60) and (59), we get that

Now, since ( x t h , a t h , I t h ) t ∈I ( k ) are i.i.d., we have by Lemma I.3 that there is an event E ′ of probability at least 1 -δ under which we have

Combining this with (65) and rearranging, we get that under E reg ∶= E ∩ E ′ :

On the other hand, we have that for all t ∈ I ( k ) :

Plugging this into (67) and using the expressions of ˆ f and f ⋆ , we get

E.4 Putting It All Together

By combining (48), (52), (53), and (49), we have that under the event E ∶= E cov ∩E reg (where E cov and E reg are as in Lemma G.1 and Lemma E.1, respectively):

Thus, plugging in the expression of T 0 from Algorithm 3, and ignoring polynormal factors d, A, H, log (∣ Φ ∣ ε -1 δ -1 ) , we get that

where the last step follows by setting ν = T -1 / 5 .

F Proof of Theorem 4.2 (Model-Free, Bandit Feedback, Adaptive Adversary)

We let I ( k ) denote the rounds in the k th epoch:

where T 0 is as in Line 8 of Algorithm 4. Throughout the analysis, we condition on the event

where E cov , E rep + span , and E freed are as in Lemma G.1, Corollary F.1, and Lemma F.3, respectively. We start our analysis by applying the performance difference lemma.

Applying the performance difference lemma. For any k ∈ [ K ] , t ∈ I ( k ) , and let ρ ( k ) be the distribution of the random policy:

Thus, by the triangle inequality, we have

We start by bounding the first term on the right-hand side of (74) (the regret term).

F.1 Bounding the Regret Term

Fix h ∈ [ H ] and define ˜ π ⋆ h (⋅ ∣ x ) ∶= ( 1 -γ / T ) ⋅ π ⋆ h (⋅ ∣ x ) + γπ unif (⋅ ∣ x )/ T for all x , where γ is as in Algorithm 4. We have that ∣ ̂ Q ( k ) h ( x, a )∣ = ∣ ¯ ϕ rep h ( x, a ) ⊺ ˆ θ ( k ) h ∣ ≤ 8 Hd 2 (since ˆ θ ( k ) h ∈ B 2 d ( 4 Hd 2 ) and ∥ ¯ ϕ rep h ( x, a )∥ ≤ ∥ ϕ loss h ( x, a )∥+∥ ϕ rep h ( x, a )∥ ≤ 2 ). By applying I.5, we have that for any η ≤ 1 8 Hd 2 and x ∈ X :

F.2 Bounding the Bias Term

To bound the bias term (second term in (74)), we make use of the following result.

Lemma F.1. For t ∈ [ T ] , h ∈ [ H ] , and π ∈ Π , there exists ¯ θ t,π h ∈ B 2 d ( H √ d ) such that for all ( x, a ) ∈ X × A and history H t -1 = ( x 1 ∶ t -1 1 ∶ H , a 1 ∶ t -1 1 ∶ H ) ,

Proof. Fix t ∈ [ T ] , h ∈ [ H ] , and π ∈ Π . By the low-rank MDP structure and the normalizing assumption on µ ⋆ h in Assumption 2.1, there exists w t,π h + 1 ∈ B d (( H -s ) √ d ) such that for all ( x, a ) ∈ X × A and history H t -1 = ( x 1 ∶ t -1 1 ∶ H , a 1 ∶ t -1 1 ∶ H ) ,

Now, with g t h as in Assumption 4.2, (15) and (77) implies that ¯ θ t,π h ∶= [ g t h , w t,π h + 1 ] ∈ R 2 d (where [⋅ , ⋅] denotes the vertical stacking of vectors) satisfies the desired property. Fix epoch k ∈ [ K ] , round t ∈ I ( k ) , layer h ∈ [ 2 .. H ] , and π ′ ∈ Π . By Lemma F.1, there exists ¯ θ t, ̂ π ( k ) h ∈ B 2 d ( Hd 1 / 2 ) such that for all ( x, a ) ∈ X × A ,

With this, let w t, ̂ π ( k ) h + 1 ∈ B d ( 3 Hd 2 ) be as in Corollary F.1 with f ( x ) ∶= 1 Hd 1 / 2 max a ∈A ¯ ϕ ⋆ h + 1 ( x, a ) ⊺ ¯ θ t, ̂ π ( k ) h + 1 ; note that this function belongs to the function class F h + 1 in Algorithm 4. With this, we define

where g t h ∈ B d ( 1 ) is such that ℓ ( H -) = h ( ) ⊺ h

With this notation, we now bound the bias term in Eq. (74): we have

and by Corollary F.1 again (in particular (93)) and the triangle inequality

where the last inequality follows by Corollary F.1 (in particular (92)).

Next, we bound the estimation error term in the right-hand side of (79).

F.3 Bound the Regression Error

By Lemma F.3 (Freedman's inequality), we have that for all π ∈ Ψ span h and ¯ θ ∈ B 2 d ( 4 Hd 2 ) :

On the other hand, we have

Now, by Corollary F.1 (in particular (92)) and the triangle inequality, we have

Thus, by combining (80), (81), and (82), we have

Using that ˆ θ ( k ) h is the minimizer over B 2 d ( 4 Hd 2 ) of the left-hand side, and the right-hand side evaluted to dN reg ⋅ ε freed + νN reg ε rep H with ¯ θ = 1 N reg ∑ t ∈I ( k ) ¯ ϑ t, ̂ π ( k ) h ∈ B 2 d ( 4 Hd 2 ) , we have that

Now, combining this with (80) and (81) with ¯ θ = ˆ θ ( k ) k , we get that

F.4 Putting It All Together

By combining (74), (79), (83), and (75), we have that under the event E in (72):

Thus, plugging in the expression of T 0 , ε freed , and ( ε span , ε rep ) from Algorithm 4, Lemma F.3, and Corollary F.1, respectively, and ignoring polynormal factors d, A, H, log (∣ Φ ∣ ε -1 δ -1 ) , we get that

where the last step follows by setting N reg = T 2 / 3 .

F.5 Spanner Guarantee

Algorithm 7 Spanner : Computing an Approximate Spanner.

Require: Layer h , feature classes Φ , policy covers Ψ 1 ∶ H , feature map ¯ ϕ ∶ X × A → R 2 d , # of episodes n . √

2: For θ ∈ R 2 d and ( x, a ) ∈ X × A , define

1: Define G = { g ∶ ( x, a ) ↦ ϕ ( x, a ) ⊺ w ∣ ϕ ∈ Φ , w ∈ B d ( 2 d )} .

3: Set G h = {( x, a ) ↦ ¯ ϕ h ( x, a ) ⊺ θ ∶ θ ∈ B 2 d ( 1 )} , and for t ∈ [ h -1 ] , set G t = G .

4: For each t ∈ [ h ] , set P t = unif ( Ψ t ) .

5: For θ ∈ R 2 d , define LinOpt ( θ ) = PSDP ( h, r 1 ∶ h (⋅ , ⋅ ; θ ) , G 1 ∶ h , P 1 ∶ h , n ) ∈ Π . // PSDP as in

6: For θ ∈ R 2 d and π ∈ Π , define LinEst ( π ) = EstVec ( h, ¯ ϕ h , π, n ) . // EstVec as in Mhammedi et al. (2023). √

Mhammedi et al. (2023).

7: Set π 1 ∶ 2 d = RobustSpanner ( LinOpt (⋅) , LinEst (⋅) , 2 , Ad 2 log ( ndH ∣ Φ ∣/ δ ) αn ) . // RobustSpanner as in Mhammedi et al. (2023).

8: Return: Policy cover { π 1 , . . . , π 2 d } .

Lemma F.2 (Spanner Guarantee) . Let ε, α, δ ∈ ( 0 , 1 ) , h ∈ [ H ] , n ≥ 1 , and ¯ ϕ h ∶ X × A → R 2 d be given. Suppose that Assumption 2.1 and Assumption 4.1 hold, and let Ψ 1 ∶ h be such that for all s ∈ [ h ] , Ψ s is an ( α,ε ) -policy cover for layer s with ∣ Ψ s ∣ = 2 d . Then, the output Ψ span h =

Spanner ( h, Φ , Ψ 1 ∶ h , ¯ ϕ h , n ) (Algorithm 7) is such that ∣ Ψ span h ∣ = 2 d and, with probability at least 1 -δ , for all π ′ ∈ Π , there exist { β π ∈ [-2 , 2 ] ∶ π ∈ Ψ span h } such that where

where c > 0 is a large enough absolute constant. Furthermore, the number of episodes T span ( n ) used by the call to Spanner is at most ̃ O ( H 2 d 2 n ) .

Proof. To derive the desired bound, we will use the generic guarantee of RobustSpanner from (Mhammedi et al., 2023, Proposition E.1). To invoke this result, we first need to derive guarantees for the optimization and estimation subroutines LinOpt and LinEst withn the Spanner algorithm (Algorithm 7). In particular, we need to show that there is some ε ′ ∈ ( 0 , 1 ) such that (with high probability) for any ¯ θ ∈ R 2 d ∖{ 0 } and π ∈ Π , the outputs ˆ π ¯ θ ∶= LinOpt ( ¯ θ /∥ ¯ θ ∥) and ˆ ϕ π ∶= LinEst ( π ) satisfy

With this, we can apply (Mhammedi et al., 2023, Proposition E.1) to get that the output

for ε ≤ 2 ε ′ is such that for all π ∈ Π , there exist β 1 , . . . , β d ∈ [-2 , 2 ] satisfying

Since LinOpt is based on PSDP as in Line 5 of Algorithm 7 and Ψ 1 , . . . , Ψ h are ( α,ε ) -policy covers for layers 1 to h , respectively, (Mhammedi et al., 2023, Corollary H.1) implies that there is an event E PSDP of probability at least 1 -δ / 2 under which for any ¯ θ ∈ R d ∖ { 0 } , the output ˆ π ¯ θ = LinOpt ( ¯ θ ) satisfies

for a large enough absolute constant c > 0 . On the other hand, since LinEst is based on EstVec as in Line 6, (Mhammedi et al., 2023, Lemma G.3) implies that there is an event E EstVec of probability at least 1 -δ / 2 under which for all π ∈ Π , the output ˆ ϕ π ∶= LinEst ( π ) satisfies

for a large enough absolute constant c > 0 . Therefore, under E PSDP ∩ E EstVec , LinOpt and LinEst satisfy (90) with

Therefore, by (Mhammedi et al., 2023, Proposition) and the fact that d √ A log ( ndH ∣ Φ ∣/ δ ) αn ≤ ε ′ , the output

is such that for all π ∈ Π , there exist β 1 , . . . , β 2 d ∈ [-2 , 2 ] satisfying

where ε span ( n, α, δ ) is as in (89).

Bounding the number of episodes By (Mhammedi et al., 2023, Proposition E.1), RobustSpanner calls LinOpt and LinEst as most ̃ O ( d 2 ) times. Each call to LinOpt [resp. LinEst ] requires H 2 n episodes. This implies the desired bound on the number of iterations.

F.6 Representation + Spanner

Corollary F.1. Let ε, δ ∈ ( 0 , 1 ) , ϕ rep 1 ∶ H , and Ψ span h be as in Algorithm 4. Then, for all h ∈ [ H ] , ϕ rep h ∈ Φ and ∣ Ψ span h ∣ = 2 d and there is an event E rep + span of probability 1 -3 δ / 4 under which for all h ∈ [ H ] :

Proof. From Algorithm 4, we have

ϕ rep h = RepLearn ( h, F h + 1 , Φ , unif ( Ψ cov h ) , T rep ) and Ψ span h = Spanner ( h, Φ , Ψ cov 1 ∶ h , ¯ ϕ rep h , T span ) , where

and α ∶= 1 8 Ad . By Lemma G.1, there is an event E cov of probability at least 1 -δ / 4 under which, for all h ∈ [ H ] , Ψ cov h is an ( α,ε ) -policy cover for layer h with ∣ Ψ cov h ∣ = d . In what follows, we condition on E cov . By Lemma G.2 and Lemma F.2, there are events E rep and E span of probability at least 1 -δ / 4 each such that under E rep ∩E span (92) and (93) hold; this follows from (98) and (88) and the choices of T rep and T span in (94). Finally, by the union bound, we have P [E cov ∩E rep ∩E span ] ≥ 1 -δ which completes the proof.

F.7 Martingal Concentration

Lemma F.3. Let K , N reg , ¯ ϕ rep 1 ∶ H , and I ( k ) be as in Algorithm 4 for k ∈ [ K ] . There is an event E freed of probability at least 1 -δ / 4 under which for all ¯ θ ∈ B 2 d ( 4 Hd 2 ) , h ∈ [ H ] , k ∈ [ K ] , and π ∈ Ψ span h :

where the random variables h t , ζ t , π t , and H t -1 are as in Algorithm 7.

Proof. Fix ¯ θ ∈ B 2 d ( 4 d 2 ) , h ∈ [ H ] , k ∈ [ K ] , and π ∈ Ψ span h . We apply Lemma I.2 (Freedman's inequality) with

R = 4 Hd 2 ;

n = N reg ;

The random variable w i set as the difference

The filtration F i set as the σ -algebra σ ( H t i -1 ) ;

The variance term V n has the following upper bound

to get that there is an event E freed h,k,π ( ¯ θ ) of probability at last 1 -( N reg ) -d H -1 K -1 d -1 δ / 8 under which

Let C be a minimal ( dHN reg ) -1 -cover of B 2 d ( 4 Hd 2 ) with respect to the ∥ ⋅ ∥ distance. Under the event

Eq. (96) holds for all h ∈ [ H ] , k ∈ [ K ] , and ¯ θ ∈ B 2 d ( 4 Hd 2 ) up to an additive O ( 1 / N reg ) error. By the union bound, we have P [E freed ] ≥ 1 -δ / 4 which completes the proof.

G Policy Cover and Representation Learning Algorithms

In this section, we present guarantees for VoX , RepLearn , and RobustSpanner which we need in the analysis of our oracle efficient algorithm. The results are based on (Mhammedi et al., 2023).

G.1 Policy Cover

The following result is a restatement of (Mhammedi et al., 2023, Theorem 12).

LemmaG.1 (VoX Guarantee) . Let ε, δ ∈ ( 0 , 1 ) be given. Suppose Assumption 2.1 and Assumption 4.1 hold. Then, there is an event E cov of probability at least 1 -δ under which the output Ψ cov 1 ∶ H = VoX ( Φ , ε, δ ) is such that for all h ∈ [ H ] :

Ψ cov h is a ( 1 8 Ad , ε ) -policy cover for layer h ;

∣ Ψ cov h ∣ ≤ d .

Furthermore, the number of episodes T cov ( ε ) used by the call to VoX is bounded by ̃ O ( Ad 13 H 6 log ( Φ / δ ))/ ε 2 .

G.2 Representation Learning

Lemma G.2 (Representation Learning Guarantee) . Let ε, α, δ ∈ ( 0 , 1 ) , h ∈ [ H -1 ] , and n ≥ 1 be given and define the function class

Further, let Ψ be an ( α,ε ) -policy cover for layer h with ∣ Ψ ∣ = d , and suppose Assumption 2.1 and Assumption 4.1 hold. Then, with probability at least 1 -δ , the output ¯ ϕ rep h = RepLearn ( h, F h + 1 , Φ , unif ( Ψ ) , n ) is such that for all f ∈ F h + 1 there exists w f h + 1 ∈ B d ( 3 d 3 / 2 ) such that:

where c > 0 is a large enough absolute constant. Furthermore, the number of episodes T rep ( ε ) used by the call to RepLearn is equal to n .

Proof. By (Mhammedi et al., 2023, Theorem F.1) and the assumption that ∣ Ψ ∣ = d , there is an event E of probability at least 1 -δ / 2 under which ϕ rep h satisfies:

where c is a large enough absolute constant. We use this to show (98). In what follows, we condition on E . Fix π ∈ Π and f ∈ G and let w f h + 1 be the vector w ∈ B d ( 3 d 3 / 2 ) achieving the infimum in (99) for the given choice of f . Let X h,ε be the set of ε -reachable states at layer h as defined in (50). With this this, we have for all h ∈ [ H ] ,

where the last inequality follows by Lemma I.1.

We now bound the first term on the right-hand side of (100). By Jensen's inequality, we have

and so using that Ψ is a ( α,ε ′ ) -policy cover (see Definition 4.1), we have

where the last step follows by (99). Combining this with (100) yields (98).

H Lower Bound for Bandit feedback with Unstructured Losses

In full-information, one does not require any structure on the losses. We show that this is not the case for the bandit case via a lower bound depending polynomially on the number of states. This lower bound implies that the low-rank transition structure with unstructured losses does not give any significant improvements over the tabular setting.

Theorem H.1. There exists a low-rank MDP with S states, A actions and sufficiently large time step T with unstructured losses such that any agent suffers at least regret of Ω ( √ SAT ) .

Proof. We assume 4 S < √ T . The construction is an H = 1 (i.e. contextual bandit) MDP with uniform initial distribution over states. Each state is a copy of an A -armed bandit problem with Bernoulli losses with mean 1 2 , and one randomly chosen optimal arm with mean 1 2 -∆ . Following the standard lower bound construction for bandits (Lattimore and Szepesvári, 2020), there exists ∆ = Θ ( 1 / √ TAS ) such that the the regret of playing any individual bandit problem for N ≤ 2 T / S rounds is lower bounded by Ω ( N ∆ ) . Let denote N ( s ) the number of time the agent receives the initial state s , then any agent suffers a regret lower bound in our MDP of

N ( s ) is the sum of T Bernoulli random variables with mean 1 / S . We have by Hoeffding's inequality

This allows to upper bound the tail

Hence E [ min { N ( s ) , 2 T / S }] = T / S -E [ N ( s ) I ( N ( s ) > 2 T / S )] ≥ T /( 2 S ) and the regret in the MDP is lower bounded by Ω ( √ TSA ) .

I Helper Results

Lemma I.1. Let ε, B > 0 and h ∈ [ 2 .. H ] be given. For any function f ∶ X → [B,B ] and π ∈ Π , we have where

denotes the set of states that are ε -reachable at layer h .

Proof. Fix f ∶ X → [B,B ] and π ∈ Π . Using the definition of X h,ε in (102), we have that x / ∈ X h,ε only if d π h ( x ) < ε ∥ µ ⋆ h ( x )∥ . Using this, we have

where the last step follows by the normalizing assumption on µ ⋆ (see Assumption 2.1) and (Mhammedi et al., 2023, Lemma I.3).

I.1 Martingale Concentration and Regression Results

Lemma I.2. Let R > 0 be given and let w 1 , . . . w n be a sequence of real-valued random variables adapted to filtration F 1 , /uni22EF , F n . Assume that for all i ∈ [ n ] , w i ≤ R and E [ w i ∣ F i -1 ] = 0 . Define S n ∶= ∑ n i = 1 w i and V n ∶= ∑ n i = 1 E [( w i ) 2 ∣ F i -1 ] . Then, for any δ ∈ ( 0 , 1 ) and λ ∈ [ 0 , 1 / R ] , with probability at least 1 -δ ,

We now state two helpful results from Mhammedi et al. (2024b) without a proof.

Lemma I.3. Let B > 0 and n ∈ N be given. abstract set. Further, let Q⊆{ g ∶ X × A → [ 0 , B ]} be a finite function class and ( x 1 , a 1 , ε 1 ) , . . . , ( x n , a n , ε n ) be a sequence of i.i.d. random variables in X × A × R . Then, for any δ ∈ ( 0 , 1 ) , with probability at least 1 -δ , we have

where ∥ g ∥ 2 ∶= ∑ i ∈[ n ] E [ g ( x i , a i ) 2 ∣ F i -1 ] and ∥ g ∥ 2 n ∶= ∑ n i = 1 g ( x i , a i ) 2 .

Lemma I.4 (Generic regression guarantee) . Let B > 0 , n ∈ N , and f ⋆ ∶ X × A → [ 0 , B ] be given. Further, let F ⊆ { f ∶ X × A → [ 0 , B ]} be a finite function class and ( x 1 , a 1 , ε 1 ) , . . . , ( x n , a n , ε n ) be a sequence of i.i.d. random variables in X × A × R . Suppose that

f ⋆ ∈ F ;

b 1 , . . . , b n ∈ R (not necessarily i.i.d.);

z i = f ⋆ ( x i , a i ) + ε i + b i , for all i ∈ [ n ] ;

ε i ∈ [-B,B ] , for all i ∈ [ n ] ; and

E [ ε i ∣ x i , a i ] = 0 .

Then, for ˆ f ∈ argmin f ∈F ∑ n i = 1 ( f ( x i , a i ) -z i ) 2 and any δ ∈ ( 0 , 1 ) , with probability at least 1 -δ / 2 , where ∥ ˆ f -f ⋆ ∥ 2 n ∶= ∑ n i = 1 ( ˆ f ( x i , a i ) -f ⋆ ( x i , a i )) 2 .

Proof. Fix δ ∈ ( 0 , 1 ) and let ̂ L n ( f ) ∶= ∑ n i = 1 ( f ( x i , a i ) -z i ) 2 , for f ∈ F , and note that since ˆ f ∈ argmin f ∈F ̂ L n ( f ) , we have where ∇ denotes directional derivative. Rearranging, we get that

Thus, rearranging, we get

We now bound the first term on the right-hand side of (110). For this, we apply Lemma I.2 with w i = ε i ⋅ ( ˆ f ( x i , a i ) -f ⋆ ( x i , a i )) , R = B 2 , λ = 1 /( 8 B 2 ) , and F i = ∅ , and use

the union bound over f ∈ F ; and

the facts that E [ ε i ∣ x i , a i ] = 0 ,

to get that with probability at least 1 -δ / 2 ,

Combining this with (110), we get that with probability at least 1 -δ / 2 ,

This completes the proof.

I.2 Online Learning

The following is the standard guarantee of exponential weights (e.g. Lemma F.4 of Sherman et al. (2023b)).

Lemma I.5 (Exponential Weights) . Given a sequence of loss functions { g t } T t = 1 over a decision set Π , { p t } T t = 1 is a distribution sequence with p t ∈ ∆ ( Π ) , ∀ t ∈ [ T ] such that

If p 1 is a uniform distribution over ∣ Π ∣ and ηg t ( π ) ≥ -1 for all t ∈ [ T ] and π ∈ Π . Then

I.3 Reinforcement Learning

The following is standard simulation lemma which is first proposed by Abbeel and Ng (2005).

Lemma I.6 (Simulation Lemma) . For two finite-horizon MDPs ̂ M = {X , A , ℓ, { ̂ P h } H h = 1 } and M = {X , A , ℓ, { P h } H h = 1 } with horizon H and ∥ ℓ ∥ ∞ ≤ 1 . Let the corresponding value function be ̂ V π h ( x ; ℓ ) and V π h ( x ; ℓ ) for step h ∈ [ H ] . For any policy π ∶ X → ∆ (A) , we have

The following is the standard performance difference lemma which is first proposed by Kakade and Langford (2002).

Lemma I.7 (Performance Difference Lemma) . For a finite-horizon MDPs M = {X , A , ℓ, { P h } H h = 1 } starting at x 1 , and two policies π, π ′ ∶ X → ∆ (A) , we have

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes] .

Justification: The main claims made in the abstract and introductio accurately reflect the paper's contributions and scope. The claims are validated by detailed proofs.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes] .

Justification: The paper discuss the limitations of the work in the discussion section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes] .

Justification: The paper provides detailed assumptions and proofs.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA] .

Justification: This is a theoretical paper.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: This paper does not include experiments.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA] .

Justification: This paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA] .

Justification: This paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA] .

Justification: This paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes] .

Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This is a theoretical work. There is no societal impact of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA] .

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA] .

Justification: This paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA] .

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA] .

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA] .

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies online learning in low-rank MDPs with adversarial losses (having linear structure), and derives a set of regret guarantees:\n\n1. A model-based and inefficient algorithm ensures $T^{2/3}$ regret, assuming the loss feature vector is unknown. The algorithm has a two-stage design: the first stage is an initial reward-free exploration stage from [1] to ensure low estimation error for the second (exploitation) stage.\n\n2. A model-free and oracle-efficient algorithm ensures $T^{5/6}$ regret, assuming the loss feature vector is unknown. This is thanks to recent advances in exploration for low-rank MDP by [2].\n\n3. A model-free and oracle-efficient algorithm ensures $T^{5/6}$ regret against adaptive adversaries, assuming the loss feature vector is known, by leveraging a representation learning algorithm from [2].\n\nSTRENGTHS:\nBy adopting recent progress in representation learning and low-rank MDP from the literature, the paper provides a set of results for low-rank MDPs with adversarial (linear) losses under bandit feedback. This is the first set of results for the more challenging bandit feedback.\n\nWEAKNESSES:\nI overall appreciate the contributions from this work, in which the authors take advantage of SOTA techniques in the literature, but also with some original adaptations, to give a complete picture on regret bounds in a new setup. However, I think the author could do a better job in presentation and delivering insights, which may make readers benefit more from reading this paper. Below are some questions that I think should be made clear(er) in the paper (and I don’t get an answer after reading the paper):\n\n1. Maybe I missed it somewhere, but what is exactly the “oracle” that is referred to throughout this paper? Is it hidden in the VoX algorithm?\n\n2. Even with linear loss struction, the regret still has to be $\\Omega(\\sqrt{A})$, is this a common result in the low-rank MDP literature, or unique due to the adversarial losses (and bandit feedback)?\n\n3. In Thm. 4.1, is $\\delta$ the failure probability for the regret guarantee? In other words, is Thm. 4.1 stating a high-probability regret bound? If so, why is the way of stating a high-prob. bound in Thm. 4.1 different from that in Thm. 5.1? If not, then the use of notation $\\delta$ is not consistent in the paper.\n\n4. What’s the difficulty of obtaining regret bounds against adaptive adversaries, without knowing the loss feature map? In section 5, the sentence “Given the difficulty of this setting, we make the additional assumption” fails to yield a smooth transition, as I don’t get the connection between the difficulty (it’s not even clear what it is) and the need for the assumption.\n\n5. Does the difficulty lie in obtaining high-prob regrets against oblivious adversaries? Is it true that with the help of the known loss feature, one can establish high-prob. regret bound, which easily implies regret bounds against adaptive adversaries?\n\n6. If algorithm 4 is mostly from other papers and there’s no novel modification, the authors may consider moving it to appendix to save some space in the main body for more informative messages.\n\n7. In line 10 of algorithm 1, “Define $\\sum_h^t = \\sum$”, anything missing before the $=$?\n\n8. Sketched proofs in the main body for deriving regret bounds would be very helpful, but I fully understand it due to the page limit.\n\n9. If a reference has been accepted to a conference, the authors may want to cite the conference version rather than arXiv, unless there's a significant update.\n\nI will consider increasing my rating to 6 and supporting acceptance if the authors can address these questions so that I can have a clearer understanding of the results.\n\nQUESTIONS:\nQuestions here are mainly for curiosity and will not affect my recommendation/rating much.\n\n1. Typically, such two-stage design (initial exploration for warm-up + standard online learning) may not give rate-optimal guarantee. In adversarial linear MDPs, [3] performs estimation on-the-fly to get the first rate-optimal regret. I was wondering what would be the difficulty of doing similar things here (and enjoying better regrets). Could the authors point out some other places where the sub-optimality comes from?\n\n2. In Thm. 4.1, the regret scales with the size of the function class. I think it makes some sense but any similar results in the literature?\n\nReferences\n\n[1] Cheng, Yuan, Ruiquan Huang, Yingbin Liang, and Jing Yang. \"Improved Sample Complexity for Reward-free Reinforcement Learning under Low-rank MDPs.\" In The Eleventh International Conference on Learning Representations. 2022.\n\n[2] Mhammedi, Zakaria, Adam Block, Dylan J. Foster, and Alexander Rakhlin. \"Efficient Model-Free Exploration in Low-Rank MDPs.\" arXiv preprint arXiv:2307.03997 (2023).\n\n[3] Liu, Haolin, Chen-Yu Wei, and Julian Zimmert. \"Towards Optimal Regret in Adversarial Linear MDPs with Bandit Feedback.\" In The Twelfth International Conference on Learning Representations. 2023.\n\nLIMITATIONS:\nIn the checklist, it says \"The paper discuss the limitations of the work in the discussion section\", but I didn't find a section named \"discussion\".\n\n====== update after author-reviewer discussion ======\n\nDuring the initial rebuttal and the discussion phase, the authors addressed my clarification question, so as of now I increase my score from 5 to 6", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies adversarial Low-Rank MDPs with unknown transition and bandit feedback. The authors give three main results, targeting either tighter regret or computational efficiency. The authors show that the linear structure of the reward function is necessary for the case of bandit feedback to achieve regret without dependence on the number of states.\n\nSTRENGTHS:\n1. The paper addresses a novel and challenging problem of adversarial Low-Rank MDPs with unknown transition and bandit feedback. The problem is well-motivated and has practical implications in real-world applications.\n2. The theoretical results are strong and the analyses are sound. The authors provide a comprehensive analysis of the proposed algorithms and prove the theoretical guarantees. The authors combine several techniques to handle the unknown transition and bandit feedback, which is non-trivial.\n3. The paper provides a lower bound to show we could not gain too much from a low-rank transition structure compared with tabular MDPs. This lower bound is insightful and provides a clear motivation for the linear reward structure in the bandit feedback setting.\n\nWEAKNESSES:\n1. The computational cost of the proposed algorithms is huge, even for the oracle efficient algorithms. The authors should provide more discussions on the computational complexity of the proposed algorithms.\n2. Though this paper is the first work to study adversarial Low-Rank MDPs with unknown transition and bandit feedback, the $O(T^{5/6})$ regret bound is not very satisfying. Moreover, the results have a linear dependence on the number of actions, which is not ideal.\n\nQUESTIONS:\n1. Is polynomial dependence on the number of actions necessary? Can the authors provide some intuition about the reason or discuss the possibility of improving the dependence on the number of actions?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

comments: SUMMARY:
This work initiates the study on learning adversarial low-rank MDPs with bandit feedback. The authors propose an inefficient algorithm with $T^{2/3}$ expected regret, and an oracle-efficient algorithm with $T^{5/6}$ expected regret. Further, the authors also show an oracle-efficient algorithm with $T^{5/6}$ high probability regret.

STRENGTHS:
1.	This work further advances the understanding of learning adversarial MDPs with general function approximation and the algorithmic designs are of interest.

WEAKNESSES:
1.	The first algorithm is generally hard to be implemented due to its computational intractability and thus might be less appealing to practitioners in the area. Besides, the algorithmic design of the first algorithm is similar to those of [1,2] in the sense that learning adversarial MDPs is somewhat reduced to learning adversarial bandits by running exponential weights over the policy set of the corresponding MDP. I would suggest the authors give more comparisons and discussions between the design of the first algorithm and the algorithms in [1,2].

[1] Kong et al. Improved Regret Bounds for Linear Adversarial MDPs via Linear Optimization. TMLR, 24.

[2] Liu et al. Towards Optimal Regret in Adversarial Linear MDPs with Bandit Feedback. ICLR, 24.

QUESTIONS:
1.	In Eq. (12), log-barrier instead of the common Shannon entropy regularizer is used in FTRL. Is this due to that it is not feasible to bound the stability term in the analysis of FTRL when using Shannon entropy caused by the possibly very large magnitude of the constructed $Q$-function estimates?
2.	Intuitively, what is the reason that Algorithm 2 still needs to operate in epochs after the phase of reward-free exploration?

LIMITATIONS:
Not applicable.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper explores regret minimization in low-rank MDPs with adversarial bandit loss and unknown transitions. This means that the transition can be expressed as $P(x' \\mid x,a ) = \\phi^\\star (x,a)^T\\mu^\\star(x')$ for some unknown  $\\phi^\\star$ and $\\mu^\\star$ and the loss is also linear in $\\phi^\\star$. They consider three distinct settings:\n\n1.  **Model-based setting**: Here, the authors assume access to function classes containing $\\mu^\\star$ and $\\phi^\\star$. They use some reward-free algorithm to obtain $\\mu$ and $\\phi$ that approximates the dynamics well, and then run EXP2-style algorithm on some $\\epsilon$-cover of the class of linear policies. This algorithm is computationally inefficient and achieves $T^{2/3}$ regret,  with some logarithmic dependencies on the size of the function classes.\n    \n2. **Model-free setting**: In this setting, they don't assume access to a class that contains $\\mu^\\star$. Once again, the algorithm begins with some of-the-shelf exploration phase that aim to find a policy cover. That is, a (small) set of policies that guarantees some sense of minimal reachability to all reachable states. Next, they run an FTRL-style algorithm in each state with log-barrier regularization on blocks of certain size. Within each block, the algorithm plays the policy derived from the FTRL, mixed with a uniform policy from the policy cover. The policy cover's exploration guarantee allows for good least-squares estimation of the Q-function for the policy employed in that block, subsequently used in the FTRL. This achieves $T^{5/6}$ regret against oblivious adversary.\n    \n3.  **Model-free setting with Adaptive adversary**: Here, the adversary adjusts their strategy based on the learner's actions, and there is additional assumption that the the loss features are known (and the dynamics feature map is still unknown). The algorithm is similar to the algorithm in 2. but adds some representation learning algorithm on top of the reward-free algorithm which suggested to be beneficial against adaptive adversary. The regret bound obtained in this setting is also of order of $T^{5/6}$.\n\nSTRENGTHS:\n-   Low-rank MDPs and representation learning are important areas of study in reinforcement learning, particularly due to the relevance of representation learning to deep RL. The paper advances research in this regime by extending previous results from full-information settings to bandit feedback.\n    \n- The paper introduces three algorithms, each tailored to a distinct setting. These algorithms manage to achieve similar regret guarantees to previous works that dealt with the full-information case (although the settings are not directly comparable). Additionally, the authors provide a compelling justification for why they cannot handle arbitrary (non-linear) loss as in the full-information case by presenting a lower bound that scales with the number of states.\n\nWEAKNESSES:\n-   The approach taken in Algorithm 1 is somewhat generic, employing an off-the-shelf reward-free exploration phase followed by an inefficient EXP2 algorithm. Most of the complexity introduced by the unknown feature maps is largely addressed by leveraging existing reward-free algorithms, possibly underplaying the novelty or the specific challenges directly tackled by the new algorithm itself.\n    \n-   Section 5 lacks detailed explanations. The section does not adequately clarify what specifically becomes more challenging in the adaptive adversary setting, nor does it thoroughly explain how the spanner policies effectively address these challenges. The description merely states that these policies are necessary *\"because the estimation of the Q-functions becomes much more challenging with an adaptive adversary\"*. This lack of detail leaves the reader without a clear understanding of the problem mechanics or the solution’s efficacy in this context.\n\nQUESTIONS:\nCould you please clarify how (41) is obtained from Lemma G.5? (41) bounds the average distance, while, to my understanding, G.5 bounds the distance on the data. It feels like some argument on the generalization gap is missing.\n- Is there a specific reason to use log-barrier and not negative entropy regularization in algorithm 2?\n\nLIMITATIONS:\nAs far as I'm concerned, the authors address most of their limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}