Reinforcement Learning in Linear MDPs: Constant Regret and Representation Selection

Abstract: We study the role of the representation of state-action value functions in regret minimization in finite-horizon Markov Decision Processes (MDPs) with linear structure. We first derive a necessary condition on the representation, called universally spanning optimal features (UNISOFT), to achieve constant regret in any MDP with linear reward function. This result encompasses the well-known settings of low-rank MDPs and, more generally, zero inherent Bellman error (also known as the Bellman closure assumption). We then demonstrate that this condition is also sufficient for these classes of problems by deriving a constant regret bound for two optimistic algorithms (LSVI-UCB and ELEANOR). Finally, we propose an algorithm for representation selection and we prove that it achieves constant regret when one of the given representations, or a suitable combination of them, satisfies the UNISOFT condition.


Reviews and author responses:

The authors study the constant regret bound in low-rank MDP setting [Jin et al., 2020] and zero inherent Bellman error setting [Zanette et al., 2020b]. On the upper bound side, this paper shows that ELEANOR and LSVI-UCB have constant regret bounds under the gap assumption (assumption 3) and UNISOFT assumption (assumption 4) respectively. On the lower bound side, UNISOFT is shown to be a necessary assumption for the constant regret. In addition, given a set of good representations $\Phi$, the new proposed LSVI-LEADER algorithm suffers a constant regret which scales with $|\Phi|$.  The main motivation and contribution of this work is to extend the constant regret bounds (Hao et al. [2020], Papini et al. [2021]) to the MDP settings. I believe the question is interesting and the results are new to me. The paper has a good structure and is easy to follow. The proof looks good from a quick overview. I have some detailed comments as below:

1.	The constant regret bounds are quite interesting results. The intuition is the algorithm will learn a lot each time when it makes an error due to the gap assumption. So eventually the algorithm could learn the environment accurately enough to obtain a near-optimal policy. Again, by the gap assumption, this near-optimal policy would be the optimal policy.
2.	For the representation selection part, though the regret bounds are constant, I wouldn’t say they are desired. Since the feature class could be quite large and exponential in general, we usually only want a polylog($|\Phi|$) bound. However, the cost in this paper would be $|\sqrt \Phi|$.
3.	The LSVI-LEADER doesn’t seem to be computationally efficient since it enumerates the whole feature class, and the optimization problem for finding min over $\phi\in\Phi$ for the $\overline Q$ seems to be hard. I suggest the authors to add related discussions in the subsequent version.
4.	In the representation selection part, only low-rank MDPs are investigated, and the reason mentioned in the paper is that the authors want to focus on the computationally tractable algorithm LSVI-UCB. However, as discussed in the last bullet, the LSVI-LEADER doesn’t appear to be computationally efficient as well. Therefore, I think this argument is not very convincing to me. Have you thought about representation selection extension for ELEANOR? If so, what is the difficulty?
5.	This paper only focuses on the exact realizable case, but the original LSVI-UCB and ELEANOR can both handle approximate setting. Do the constant regret bounds also hold for the approximately case?
6.	Maybe I’m missing something, can the authors explain why span(u_i) = span(\phi_j) in Line 857?
7.	In line 23, the discussion about realizability is not very clear. The exponential sample complexity refers to the exponential blow up in the approximation error. However, in the exact realizable case, we will not suffer from the exponential sample complexity or regret bounds. If you allow an additive term to be sqrt d larger than the approximation error, then actually poly results are achievable.
8.	Line 110, “it implies the realizability of any value function” is not very clear. I think the point here is for any value function, its Bellman backup is linear in the feature. 
9.	I find the term learnability/learnable quite confusing. In the footnote on page 1, learnable representation broadly refers to the representation where finding near-optimal policy is a learnable problem. However, it might let people think the representation is able to be learnt by the algorithm. 
10.	The constant $c_M$ in line 182 is not clear. It would be better to mention what parameters are treated as constant and included in $c_M$.
11.	Line 327 . is missing before “With”.


Post rebuttal:
Thanks the authors for addressing my questions.

After the long discussions and reading other reviews, I intend to degrade my score to 5.

I appreciate the constant regret result in the known feature case. The Unisoft assumption seems to be reasonable because the authors provide a necessary result.

I also have several major concerns about this paper. (i) This paper makes an assumption that the optimal policy is unique (as acknowledged by the authors). And this assumption is much stronger than the gap assumption. (ii) The constant regret bound hides many crucial factors. For example, the factor could be exponentially small, which makes the constant regret bound exponentially large. I'm not that satisfied with the constant bound if it scales exponential with standard parameters. I think to get a meaningful polynomial constant regret bound, the authors need to make additional assumptions. (iii) Similar as (ii), the constant regret bound in feature selection case scales polynomial with the number of features, while we usually just want to pay log in this factor. In this sense, I don't feel the algorithm is doing "representation selection". N/A

 Hello again reviewer aaaf, we would be grateful if you can confirm whether our responses has addressed your concerns, and let us know if any issues remain. 

Thank you in advance. 


 Hello reviewer aaaf, we would be grateful if you can confirm whether our responses has addressed your concerns, and let us know if any issues remain. 

Thank you in advance. 

 Dear Reviewer aaaf,
 
Thank you for the follow-up. 

As mentioned in our response, we do agree that the polynomial dependence on the number of representations is bad, and whether that could be reduced to a logarithmic dependence remains our key open question for this problem of selecting “good” representations (we note that the same open question was raised in the concurrent work of Zhang et al. 2021). However, we would like to stress that we believe that a comparison with Modi et al. would be orthogonal to ours. Here are a few additional arguments on this matter:

1. Our representation selection objective is different (and arguably more challenging) than the one considered by Modi et al. In particular, Modi et al. aim at finding a representation with low mean square error (i.e., one that induces a low-rank MDP). On the other hand, we wish to find a UNISOFT representation among a set of realizable representations. That reducing the MSE is not enough for this purpose was already shown by Papini et al.: in their Appendix F, where they deal with misspecification, an MSE-based test is used to eliminate misspecified representations. It is shown that this only allows the algorithm to end up with a set of realizable representations, but after that a different algorithmic scheme, whose primary objective is reducing regret  (like LSVI-LEADER), is needed to find the UNISOFT one. Therefore, even if the approach of Modi et al. could be extended to the regret minimization setting with a logarithmic dependence on $|\Phi|$, there would be no guarantee that running that algorithm would recover a UNISOFT representation as in our case. Hence the question on how to achieve this with a $\log|\Phi|$ dependence remains open.

2. The reviewer suggests that “The sample complexity guarantee provided by Modi et al. can be transformed to a sublinear regret guarantee”. Still, that is not a result provided by Modi et al. and it is not clear to us how to obtain it. In particular, it is not just a matter of translating a sample complexity bound into a regret bound: both the interaction protocol and the algorithmic schemes are different w.r.t. our work. Even if we modified the algorithm of Modi et al. to work in a setting where the reward is given (e.g., best-policy identification, which is probably the most trivial thing to do), how to extract a result that is comparable to ours is not clear. We would first need to make the algorithm of Modi et al. work in the regret minimization setting, analyze it, and finally compare the results. But that is a completely different approach to the one they take. Hence, we would be comparing to results that do not exist in the literature.

3. Even if we directly translated the sample complexity bounds of Modi et al. into regret bounds, we note that, while it is true that they could scale as $\log|\Phi|$, they also contain several dependencies which are orders of magnitude worse than in our work. For instance, the sample complexity provided in their Theorem 2 scales as K^13 (K is the number of actions). This is also an unreasonable dependence in any case of practical interest we can think of . Thus, it is not clear whether there will be overall an improvement in the final bound compared to ours.

4. Last but not least, we remark that the paper of Modi et al. appeared on arxiv on Feb 14th 2021 and it is also under review at this conference (at least this is what we deduce from the primary author’s webpage and CV). Therefore, while we have cited that work and we will expand its discussion in the final version, we believe that it is unfair to base the assessment on a comparison to it since it is a concurrent paper that faces a problem orthogonal to ours.


In view of these points, we hope the reviewer could reconsider its position. We would be happy to keep discussing this or any other concern.
 
**References**

Zhang, Weitong, et al. "Provably Efficient Representation Learning in Low-rank Markov Decision Processes." arXiv preprint arXiv:2106.11935 (2021).


 Dear Reviewer tEY1,

Thank you for your additional comments. We address your new questions below.

> I still don't quite understand the argument here. For "realizability of the optimal Q function", do you mean linear Q* assumption like [1]? The references you listed in line 23 are about the approximate realizability may suffer exponential blowup in the sample complexity and is not enough. Those paper do not consider the exact realizable case as you can see Exact Linear Q∗ is shown as an open problem in table 1 of [2].

The reviewer is right, [1] is the right reference for that sentence. We will fix it. Thank you for pointing this out.
 
> I would say you at most want to pay poly in parameters like d, H, etc. So not depending on K itself might not be good enough, though it is a constant. This is also related to Reviewer WqxU's comments about the the regret bound.
 
Note that our necessity result (Theorem 5) holds for any algorithm that suffers sub-linear regret in all MDPs in $\mathcal{M}$, no matter how large is the constant $c_M$. However, as the reviewer suggests, we could indeed strengthen Definition 6 by requiring that $c_M$ is a polynomial function of relevant problem-dependent quantities (like $d$ and $H$). Then, Theorem 5 continue to hold for each algorithm satisfying this new condition (since this is a smaller class than before). Moreover, all existing “provably-efficient” algorithms we are interested in (e.g., LSVI-UCB and ELEANOR) are still included since their regret bounds are polynomial in all such problem-dependent quantities (hence their $c_M$ indeed satisfies the required property).
 
> Does the lower bound also hold if you consider deterministic linear reward, i.e. without Gaussian noise?
 
We believe it would be possible to generalize our lower bound to such a setting but that would require a more complicated construction than ours. In our proof we keep the transition probabilities of M fixed and build alternative MDPs that are hard to distinguish from M by perturbing only the rewards. For deterministic rewards that would not work (essentially because estimating the reward at some state-action (s,a) pair requires a single sample from (s,a)) and we would need to perturb the transition probabilities as well.
 
> In Assumption 4, you assumed the optimal action is unique. I think this is much stronger compared with the gap assumption, where you can have multiple optimal actions. I would suggest the authors to write assumptions separately from the UNISOFT assumption. (i) Does the UNISOFT assumption also include this part? It doesn't seem so to me. (ii) And is this assumption also necessary as the UNISOFT assumption? (iii) In the upper bound results (like Thm8 and Thm9), you mentioned that you need UNISOFT assumptions (Assumption 4). Does it mean you also need the assumption that the optimal action is unique?
 
(i) The assumption of unique optimal actions is indeed different from UNISOFT (the former is related to the MDP alone, the second is more related to the feature representation at hand). We will state the two separately as the reviewer suggests. Note, however, that the two assumptions are somewhat related since UNISOFT requires a definition of optimal features $\phi^\star_h(s)$. When multiple optimal actions exist, it is not clear which should be used to define the UNISOFT condition. In particular, we see two possibilities: (A1) we suppose that UNISOFT holds only for *some* optimal policy (i.e., with $\phi^\star_h(s) = \phi_h(s,\pi_h(s))$ where $\pi$ is an optimal policy), or (A2) that it holds for *all* optimal policies (note that a sufficient condition for this to happen is that the feature map makes optimal actions indistinguishable, i.e. $\phi_h(s,a) =\phi_h^\star(s)$ for every action a that is optimal in s). In the first case we believe it might be hard to get constant-regret upper bounds (at least with existing optimistic algorithms), while in the second case it might still be possible. See also below for the complications of analyzing the first case.
 
(ii) We believe the answer to this question really depends on how we define UNISOFT in the presence of multiple optimal actions as mentioned above. In the case of (A2), UNISOFT might be sufficient as we highlight below. In the case of (A1), we do not exclude that it might be possible to prove that constant-regret is indeed impossible in certain MDPs (at least for the kind of optimistic algorithms we study here). This remains an open question to us which we will definitely investigate in future work.
 
(iii) That is correct: we also need unique optimal actions to prove the (constant) regret upper bounds. The intuition is that, with such an assumption, we can characterize precisely what actions those optimistic algorithms converge to playing in the long run. The case of multiple optimal actions is significantly more difficult to analyze because the set of “optimal features” depends on the particular optimal policy the algorithm converges to. In summary, to get constant regret with UNISOFT we first need to show that the algorithm converges to playing certain optimal actions, and then use the fact that the features of such actions span the whole space. Unfortunately, it is an open question to characterize what policy existing optimistic algorithms converge to (and whether they converge in the first place) in the presence of multiple optimal actions. Of course, we might have no problem in proving the result in the presence of multiple optimal actions with (A2) above: since all optimal policies induce features that span the whole space, we only need to know that the algorithm plays optimally after a while (which we know for the kind of algorithms we study) without precisely characterizing which optimal policy is chosen.

Let us know if that answers your questions. We would be happy to keep discussing this or any other concern.

 Thanks for your detailed response. My main concern is still, as raised by other reviewers, the polynomial dependence on the feature class cardinality $|\phi|$. Such a polynomial dependence seems unacceptable since in most interesting application scenarios, $|\phi|$ would be large. Besides, I do not think the sample complexity guarantee and regret guarantee cannot be compared. The sample complexity guarantee provided by Modi et al. can be transformed to a sublinear regret guarantee with a logarithmic dependence on $|\phi|$. Therefore a detailed comparison between Modi et al. and your work is still needed. Because of these issues, I'm maintaining my score at 5.

 6. Now I get it. Thanks for the explanation.

7. I still don't quite understand the argument here. For "realizability of the optimal Q function", do you mean linear Q* assumption like [1]? The references you listed in line 23 are about the approximate realizability may suffer exponential blowup in the sample complexity and is not enough. Those paper do not consider the exact realizable case as you can see Exact Linear Q∗ is shown as an open problem in table 1 of [2]. 

[1] Weisz, Gellért, Philip Amortila, and Csaba Szepesvári. "Exponential lower bounds for planning in mdps with linearly-realizable optimal action-value functions." Algorithmic Learning Theory. PMLR, 2021.

[2] Du, Simon S., et al. "Is a good representation sufficient for sample efficient reinforcement learning?." arXiv preprint arXiv:1910.03016 (2019).

8. I think the value functions don't need to be the true value functions under some policy. In other words, given any function in R^S, you could back it up and get the linear form.

10.. I would say you at most want to pay poly in parameters like d, H, etc. So not depending on K itself might not be good enough, though it is a constant. This is also related to Reviewer WqxU's comments about the the regret bound.

My other points are more like comments and hope the authors would incorporate them in the next version. The constant regret bounds are a good contribution, and the most significant limitation of this work to me is poly dependence on number of features in the bound, and the computational complexity.

Additional questions: 

12. Does the lower bound also hold if you consider deterministic linear reward, i.e. without Gaussian noise?

13. In Assumption 4, you assumed the optimal action is unique. I think this is much stronger compared with the gap assumption, where you can have multiple optimal actions. I would suggest the authors to write assumptions separately from the UNISOFT assumption. (i) Does the UNISOFT assumption also include this part? It doesn't seem so to me. (ii) And is this assumption also necessary as the UNISOFT assumption? (iii) In the upper bound results (like Thm8 and Thm9), you mentioned that you need UNISOFT assumptions (Assumption 4). Does it mean you also need the assumption that the optimal action is unique? 

 Hello reviewer tEY1, we would be grateful if you can confirm whether our response has addressed your concerns, and let us know if any issues remain. 

Thank you in advance.

 Hello reviewer WqxU, thank you for answering. We would be grateful if you can confirm whether our new response has addressed your concerns, and let us know if any issues remain. 

Thank you in advance.

 As kindly requested by reviewers Aaaf and WqxU, we provide more details on the technical challenges we had to face in solving the problems studied in the paper. We hope this will help the committee appreciate the elements of novelty in our work, in particular w.r.t. previous works on contextual bandits (Papini et al. 2021) and low-rank MDPs (Jin et al. 2020, Zanette et al. 2020, He et al. 2020) that provided inspiration and fundamental building blocks for our theoretical analysis.

- Unisoft is necessary (Section 3.1/Appendix B): our proof of Theorem 5 is based on a lower bound for regret minimization in linear MDPs (Lemma 7). This approach is inspired by the proof of Proposition 2 by Papini et al. (2021) for the case of contextual bandits. However, the latter was based on an existing regret lower bound for contextual bandits by Hao et al. (2020), which does not admit an easy generalization to MDPs. Instead, our Lemma 7 is a novel lower bound for linear MDPs that was technically challenging to obtain and may also be of independent interest. 

- Unisoft is sufficient (Section 3.2/Appendix C): our proof of Theorem 19 (of which Thm 8 and 9 are special cases) follows the logic of the proof of Lemma 2 by Papini et al. (2021) on contextual bandits. The latter consists of two key steps: (1) bounding the instantaneous regret with Mahalanobis feature norms and (2) showing that these norms are decreasing under the UNISOFT assumption. Both steps raised additional technical challenges once applied to linear MDPs.
  1. Compared to contextual bandits, in linear MDPs the instantaneous regret has a more complex (recursive) relationship with feature norms, as first observed by Jin et al. (2020, Lemma B.6). We could only handle this additional challenge by deriving a novel upper bound on the gaps, that is our Lemma 18. The novelty of this result is apparent from the most general condition under which it holds (Assumption 16). “Almost local optimism” is a new algorithmic concept that is trivially satisfied by LSVI-UCB, but also less obviously by ELEANOR, for which Zanette et al. (2020) only established the weaker property of “global optimism”.
  2. Identifying the correct form of the UNISOFT assumption (i.e., the correct generalization of the HLS property by Papini et al., 2021) was crucial for establishing this result, and not a trivial task. The challenge was in correctly defining “optimal features” for the multi-step case, which turned out to be the ones observed in trajectories generated by the optimal policy. To see why leveraging the contribution of these features is technically more challenging than in the bandit case, please refer to the proof of Lemma 28, where a careful study of trajectory distributions was necessary.

- Representation selection (Section 4/Appendix D): LSVI-LEADER is conceptually similar to the LEADER algorithm by Papini et al. (2021), however its analysis comes with some additional technical challenges. As first observed by Jin et al. (2020, Lemma D.6), studying multi-step error propagation requires a precise characterization of the functional class of optimistic value functions, in terms of its covering number. The main challenge here was in handling the complex value function used by LSVI-LEADER, which is built by combining N representations (Lemma 20). This is the technical challenge that ultimately generated the polynomial dependence on N in Thm 10. We are looking into improving such dependence, but it remains an open question at the moment.

To sum up, applying ideas from contextual bandits to linear MDPs was not just a matter of combining results from the two literatures, but required original solutions that, on several occasions, pushed the theoretical understanding of the linear MDP framework.


 I thank the authors for answering my queries. As they might have understood, the concerns were more about the discussion and the way the results are presented than anything too strong or technical. It will be great if the authors do make the changes discussed in the rebuttal.

Just to clarify: I was thinking of the $SA$ dimensional features (say one-hot encoding) which are quite standard to express a tabular system linearly. In that formulation, the condition reduces to a reachability style assumption for every state-action pair, right? It was in this case, that I meant that the implicit dependence can be substantially large in terms of the horizon and other MDP parameters. Although this observation doesn't severely change the implications of the constant regret result, it seems important to consider as an example which clarifies the significance and nature of the quantity $\lambda_h^+$. It can then motivate further interesting questions of whether a weaker version of UNISOFT condition can be deemed useful.

Overall, I find the other comments satisfactory and look forward to an updated version. However, I think a common response to reviewer aaaf's question about the set of technical challenges in this work will help in a better final rating for this paper. Thanks.

 We thank the reviewer for the detailed and constructive comments, and for the time spent reviewing our paper.

Our answers follow the numbering used by the reviewer.

1. We thank the reviewer for recognizing our results as interesting. 

2. Yes, this is the main difference w.r.t. the contextual bandit setting studied by Papini et al. How to design and analyze an algorithm that adapts to the best representation (hence suffering constant regret when a UNISOFT one exists) while paying only a $log(N)$ dependence is an open question and we will investigate it in future work.

3. This is an interesting point, in particular for future generalizations to representation learning in parametric function classes like neural networks. We will add a discussion on this point.

4. Although LSVI-LEADER has some sources of inefficiency, ELEANOR is known to be intractable. The lack of local optimism of ELEANOR also makes it more difficult to analyze.

5. As observed by Papini et al. (2021) for contextual bandits, the misspecified case is very challenging. The proof relies on LSVI/ELEANOR observing optimal features often. In the misspecified setting, we completely lack a characterization of what features the algorithm will observe at $\epsilon$-optimality.

6. By linear algebra: AA^T has the same column space as matrix A, which is also the span of the eigenvector of A.

7. What we mean here is that realizability of the optimal Q function is not enough for efficiently solving an MDP.

8. What we mean here is that the value function of any policy is linear in the features, as shown by Jin et al. (2019, Prop. 2.3)

9. We acknowledge this is confusing. We will use less ambiguous definitions in the final version.

10. The constant will be specific for each algorithm. Its specific form does not matter since we are establishing a negative result. The only important thing is that it does not depend on $K$. 

11. We thank the reviewer for pointing out typos.

 We thank the reviewer for the constructive feedback and for the time spent reviewing our paper.

> I may not be able to tell whether the contribution of this work is significant without a detailed comparison between this work and Modi et al.

We think our work is orthogonal to the one by Modi et al. and similar ones. First, works like Modi et al. have representation learning as their main objective. They operate in the reward-free setting that is different from the regret minimization considered in our paper. They learn a representation that can be used to efficiently learn an optimal policy once a reward is given. For us, a reward is given from the start and learning/selecting a good representation in the meantime is just a way to suffer less regret. Second, they consider sample complexity as a performance measure while we focus on regret.
While we think the two approaches are orthogonal (and need not be compared), they could be used together in principle. First learn a set of candidate representations in a reward-free fashion, then run LEADER to select the best one(s). 
However, differences and potential synergies between the two approaches are worth mentioning in the paper, so thanks for pointing this out.

> Since the final regret depends on $\beta_k$ polynomially, it then indicates that the regret of LSVI-LEADER also depends on $N$ polynomially.

Yes, this is the main drawback compared to the contextual bandit setting. We still don’t know if this polynomial dependence can be removed but we are working on it. However, as stated above, we do not think the comparison with Modi et al. holds.

> it seems that such a regret in Theorem 10 can also be achieved by considering the following vanilla algorithm:  ...

The approach the reviewer suggests does not work in the online setting we are considering. Testing the different representations in advance requires collecting data with a suboptimal policy. How much regret would be suffered in the process? Certainly not a constant amount as LEADER does under the mixing condition. Again, this shows the fundamental difference w.r.t. Modi et al. and related works, where there is an initial “exploratory phase” where regret is not even defined (since there is no reward).


 We thank the reviewer for the positive feedback and for the time spent reviewing our paper.

> the authors need to assume that the state space is finite. 

The sufficiency results also hold for continuous state spaces, if you accept the gap assumption. As for necessity, since this is a negative result, we do not see how having infinite states could make the problem easier (without additional structure). We will clarify this in the final version of the paper.

> The UNISOFT assumption if taken into context for any tabular MDP is essentially saying that the constant regret result can be shown when the optimal policy visits each state with a minimum probability

The UNISOFT assumption does not hold for tabular MDPs. Representing a tabular MDP as a linear one requires $S\times A$ features. It is not possible to span an $SA$-dimensional space with $S$ optimal features.

> this regret bound ... feels a bit unsatisfactory as this can be exponentially large in the horizon of the MDP.

The dependence on the horizon is polynomial and is reported in Theorems 8, 9. Since we do not have an explicit expression for $\overline{\kappa}$, one may worry that it hides an exponential dependence on $H$. However, by analyzing eq. 27 in the proof of Thm. 19, we can show that $\overline{\kappa} = \tilde{O}(d^2H^4\Delta^{-2}\lambda_+^{-3})$, where $\lambda_+=\min_{h}\lambda_h^+$ and $\tilde{O}$ in this case hides logarithmic terms in $d$, $H$, $\Delta$ and $\lambda_+$ (but not in $K$). So the final dependence on $H$ is polynomial both for ELEANOR and LSVI-UCB (for the latter, $\overline{\kappa}$ only appears under the logarithm anyway).

> The authors do mention that for smaller values of $K$ , the usual $\sqrt{K}$ regret bounds might be much tighter but I think a discussion about this would be extremely important.

This simply means that, since existing upper bounds continue to hold, one can replace $\overline{\kappa}$ with $\min(K, \overline{\kappa})$ in both Thm 8 and 9. We will better emphasize this point.

> the mixing condition should be discussed more in detail with theorem 10 being a short note

Thank you for the suggestion, we will definitely add some comments to this part.

The paper investigates the necessary and sufficient properties required by the state-action representation in an MDP which lead to a constant regret in the online learning setting. Building up on the recent results on 'good' representations in stochastic linear bandits, the authors present a nice extension of the results to the MDP setting in the low inherent Bellman error and low-rank MDP case.
It is shown that the UniSOFt condition, that the features of any reachable state in the given MDP are spanned by the features of states reachable by the optimal policy, is necessary and sufficient for achieving constant regret. In addition to the unisoft condition, the authors also need to make a minimum gap assumption between the Q* values of optimal vs sub-optimal actions. For the sufficiency result, modifications of the analysis of LSVI-UCB and ELEANOR are shown to exhibit constant regret under the two assumptions. Lastly, the authors also show that if there are multiple admissible features available to the agent, the algorithm LSVI-LEADER can utilize the unisoft property if present in any of the candidates or is mixed across the candidate features.  Overall, the paper is nicely written and presents a clean and thorough analysis. The work strongly builds upon the recent results by Papini et al on constant regret and representation selection in stochastic linear bandits wherein a full-rank assumption is used on the context distribution's covariance under the optimal policy. In this paper, firstly the assumption is distilled down to a spanning assumption on any feature reachable in the given MDP and is further indexed by the set of all policies to account for the time dependence. The main challenge in the analysis is to account for the episodic structure of the MDP with multiple timesteps which now depends more on the chosen policies instead of the one-step nature of the bandit problem. Other than that, the techniques are quite 

The key representation property that the paper studies is the property that the features of any state reachable by any policy in the given MDP lie in the subspace spanned by the features seen via the optimal policy. When using this assumption with a minimum gap assumption, it is shown that the no algorithm can guarantee sub-linear regret on a family of MDPs and incur only constant regret on the given instance unless the UNISOFT condition is satisfied. The MDPs in the family all have the same transition dynamics but have different reward parameters. The proof uses a careful analysis of the properties of the feature covariance matrix and expected feature vector for the agent's sequence of policies and the optimal policy and relates it with the value gap of a policy wrt the optimal policy. Using the minimum gap assumption, it is then shown that the constant regret assumption for a given MDP can be used to upper bound the eigenvalues of the expected feature matrix seen by the agent over any $K$ episodes. However, for the results to go through, the authors need to assume that the state space is finite. This is not mentioned in the main text and seems quite important for the narrative of the paper.

The sufficiency results for LSVI-UCB and ELEANOR are quite easy to follow and are discussed well in the paper. In my run through, I couldn't find any errors in the derivations. However, one major aspect which is missing in the paper is a thorough discussion of the nature of the regret bounds which one might get with these assumptions. The UNISOFT assumption if taken into context for any tabular MDP is essentially saying that the constant regret result can be shown when the optimal policy visits each state with a minimum probability and the achievable regret would depend on the minimum visitation probability over all states ($\lambda^+$). Now, though this regret bound doesn't depend on the number of episodes and is thus constant in the MDP parameters, it feels a bit unsatisfactory as this can be exponentially large in the horizon of the MDP. If one is willing to account for such exponential dependencies, it is important to explicitly describe the nature of each term appearing in the regret bound. The authors do mention that for smaller values of $K$, the usual $O(\sqrt{K})$ regret bounds might be much tighter but I think a discussion about this would be extremely important. Also, the finiteness of the state space is also something which needs to be discussed in this context as the necessity result only applies with that assumption.

Representation selection: As mentioned before, with the analysis of the previous theorems in the paper, the selection result is quite easy to follow. I feel that the mixing condition should be discussed more in detail with theorem 10 being a short note as the dependence of $\tilde{\kappa}$ is a bit more subtle when compared to the simpler selection result. 

Concluding: Noting the details about finiteness of the state space, dependence on MDP parameters and the clarity of the result in the selection section, I'll give a weak accept rating to this paper. The paper is primarily a theoretical one and therefore doesn't have any immediate societal impact.

The authors study representation learning on either MDPs with low Bellman error or low-rank MDPs in this paper. They prove that a condition on the representation feature called UNISOFT is both necessary and sufficient for the algorithm to achieve a constant regret. They also show that a newly proposed LSVI-LEADER algorithm is able to pick up the best or mixed-best features from candidates with provable sample complexity guarantee. Finally, they prove that under the UNISOFT condition, LSVI-LEADER is able to achieve a constant regret, just as the vanilla RL setting.   To my best knowledge, this is the first work that studies the regret guarantee for representation learning. The UNISOFT condition is a nice extension of a similar condition in linear bandits literature. However, I still have some comments and questions about the current version of this work, and I hope the authors can reply to me in detail. 

In line 38, you say that previous works [Du et al., 2019, Agarwal et al., 2020, Modi et al., 2021] only study the learnable representations and do not study their sample complexity or regret. That really confuses me since Modi et al. clearly stated the sample complexity for their algorithm to find a near-optimal value function or policy (See their Theorem 3 or Corollary 4). Besides, since both Modi et al. and this work focus on the low-rank MDP setting and they all have sample complexity or regret guarantee, I think it is natural to ask which result is better by considering either their assumptions or complexity guarantee. I may not be able to tell whether the contribution of this work is significant without a detailed comparison between this work and Modi et al.


In Theorem 10, you imply that LSVI-LEADER suffers a regret that depends on the confidence value $\beta_k \sim O(\sqrt{N})$, where $N$ is the number of the representation candidates. Since the final regret depends on $\beta_k$ polynomially, it then indicates that the regret of LSVI-LEADER also depends on $N$ polynomially. That seems like a much worse dependence on $N$ compared with that in Modi et al., where the dependence is only logarithmic. Besides, it seems that such a regret in Theorem 10 can also be achieved by considering the following vanilla algorithm: we first compute $\tilde R(K, z, \beta_k)$ for each $z \in Z$, then we pick up the $z$ with the smallest $\tilde R(K, z, \beta_k)$ and run LSVI-UCB on it for $K$ episodes. If that is true, then what is the point for us to use LSVI-LEADER if we want to minimize the regret?

Meanwhile, it is hard to judge the technical contribution of this work. It seems that the tools in this work to analyze the performance of ELEANOR or LSVI-UCB are quite standard, and most of them are from previous Zanette et al., Jin et al., and He et al.. The most marvelous technical contribution part is the proposed UNISOFT condition, while itself also originates from the previous linear bandits work. I do not want to criticize its originality, since maybe for this specific problem, the adopted approach, although standard, is the only available approach. However, I truly want to see the technical challenges you faced when you solve this problem, since they are crucial especially for a pure theoretical paper. 
 They discussed the limitations. I do not see any negative societal impact on this work.

Meta-review: In this work, the authors theoretically study representation learning on MDPs with low Bellman error and low-rank MDPs. The main concerns from the reviewers are about the polynomial dependence on $N$ in the main results and the lack of detailed comparisons with the existing works. We suggest that the authors can further improve their theoretical analysis and add more detailed discussions in their next version.  

Decision: accept