Title: An Exponential Lower Bound for Linearly Realizable MDP with Constant Suboptimality Gap
Authors: Yuanhao Wang
Ruosong Wang
Sham M. Kakade
Venue: NeurIPS 2021
Keywords: Reinforcement Learning
Linear Function Approximation
Lower Bound

Abstract: A fundamental question in the theory of reinforcement learning is: suppose the optimal $Q$-function lies in the linear span of a given $d$ dimensional feature mapping, is sample-efficient reinforcement learning (RL) possible? The recent and remarkable result of Weisz et al. (2020) resolves this question in the negative, providing an exponential (in $d$) sample size lower bound, which holds even if the agent has access to a generative model of the environment. One may hope that such a lower can be circumvented with an even stronger assumption that there is a \emph{constant gap} between the optimal $Q$-value of the best action and that of the second-best action (for all states); indeed, the construction in Weisz et al. (2020) relies on having an exponentially small gap. This work resolves this subsequent question, showing that an exponential sample complexity lower bound still holds even if a constant gap is assumed.  Perhaps surprisingly, this result implies an exponential separation between the online RL setting and the generative model setting, where sample-efficient RL is in fact possible in the latter setting with a constant gap. Complementing our negative hardness result, we give two positive results showing that provably sample-efficient RL is possible either under an additional low-variance assumption or under a novel hypercontractivity assumption.


Review discussion:
The paper considers an interesting open question on the online RL with linear function approximation. Realizability and constant minimum gap are commonly used, and reasonable assumptions for this setting, but nevertheless, the paper shows that it is still a hard problem without further assumptions. This negative result is a good addition to the current literature. For the upper bound, a major contribution is to remove the linear dependency on the number of actions with a nice adaptation of G-optimal experimental design.
Overall, the paper is very well-written and presents solid improvements/contributions to the RL theory, hence I recommend the acceptance.
Main Comments
- It seems that there is a caveat in this negative result: there is no limit on the action space and thus there could be exponentially many possible actions to choose. It would have been better clarified if the same negative result can be obtained when the number of actions are finite (like in many practical problems).
- Could you explain what Assumption 4 (hypercontractivity) means in the RL world more explicitly?
Unlike in robust-statistics where the hypercontractivity is crucial for the success of specific moment-based methods, for the proposed algorithm it is hard to see why it helps to overcome the impossibility result. Is Assumption 4 some sort of uniform ergodicity? or is it some variation of (nearly) deterministic systems?
- How broadly Assumption 3 or 4 can be applied in practice? What are some practical examples whose transition kernels are not completely deterministic but still satisfying these assumptions?
Minor Comments
- Line 225 - 226: One style of assumption ... [Zanette et al., 2020] -> I don't understand how this work is relevant to your assumptions.
- Section 5.1: It might be nicer to state and emphasize what is the major challenge / target to improve at the beginning of the section.
No
We would like to thank the reviewer for the insightful and positive comments.
We agree that our hard instance requires an exponential number of actions to work, and that will be made clearer in the next version.
However, it is unclear to us whether the exponential lower bound still holds when the number of actions is polynomially small.
In fact, we spent a serious amount of time on this problem and got no luck. We also tried hard instances similar to that suggested by the reviewer and did not succeed. Given this, it might still be possible that the problem is solvable with a polynomial number of samples when the action space has constant size (or even $\mathrm{poly}(d)$ size).
Intuitively, hypercontractivity states that for any policy, features of the induced state-action distribution satisfy certain anti-concentration properties. Since Gaussian distributions (with arbitrary covariance matrices) satisfy hypercontractivity, a practical example where hypercontractivity holds is the case when the state transition has Gaussian noise (as common in control settings). We will add more discussion in the next version.
Overall I think the contribution of the paper is significant and the presentation is quite clear. To the best of my knowledge most of the related works have been sufficiently addressed.
Here are my suggestions for the paper:
* In the paper, it would help (subject to space constraints) to (i) include a figure detailing a rough description of the lower bound in the paper and (ii) a discussion comparing the current construction to the previous lower bound of Weisz et al.
* In section 4.1 it would help to add a line that the total number of states in the MDP is also equal to $m$.
* In the presentation, I sometimes got confused with notations such as $\bar{a}_1$, $\bar{a}_2$, etc. which are in fact used to refer to states and not actions. This is a bit jarring at times. I think in general it may help to only index the states by $\bar{i}$ and $\bar{j}$ and avoid using the notation $a_\cdot$ entirely here.
* In Theorem 2, it might helpful for a reader to see a more explicit description of the number of trajectories consumed by the algorithm instead of $\text{poly} (1/\epsilon)$.
* In the case the action space constrained to be $\text{poly}(d)$, is there any conjectured approach for a lower bound?
A natural approach is to split each state in the current (exponential action space) MDP into an exponential number of states with a constant number of actions each, and inducing the uniform distribution over these states upon taking any action in the previous state.
However, this approach fails because even the value of the optimal policy becomes exponentially small.
But of course, there is a lot more room to work around here.
So, I wonder: do you believe the flexibility of the next-state distribution here is sufficient to get such a lower bound instance to work when the action space is only polynomially large?
No
We would like to thank the reviewer for the insightful and positive comments.
Suggestions on the writing will be taken.
In particular, we will add a figure to describe the high-level construction of our hard instance, together with a new paragraph comparing our construction to that of Weisz et al..
It is unclear to us whether the exponential lower bound still holds when the number of actions is polynomially small.
In fact, we spent a serious amount of time on this problem and got no luck. We also tried hard instances similar to that suggested by the reviewer and did not succeed. Given this, it might still be possible that the problem is solvable with a polynomial number of samples when the action space has constant size (or even $\mathrm{poly}(d)$ size).
I thank the authors for their submission, I liked the work very much.
Strengths:
+ the work has some genuinely new ideas in the construction (although it leverages prior techniques like JL lemma)
+ it clarifies what’s achievable in the far more interesting online setting (i.e., the separation with the generative model setting)
+ the construction is easier than Weisz et al ’20, making it more amenable to explanation
I read the upper bound section a bit faster, but I’d like the authors to clarify a bit better the assumption about hypercontractivity  (in particular, its meaning in the RL setting) in the final version of the work.
The work makes a non-trivial technical contribution, but most importantly, it presents a key new result: while Weisz et al ’20 also give MDPs that are hard to learn in the online setting considered here, the construction there is more contrived / pathological (this is by necessity). By making the setting more realistic (i.e., online), learning becomes harder and the authors leverage a less pathological construction to still get ~ e^d complexity, further supporting the idea that such hardness is more ``   `realistic'.
I support the work for publication to NeurIPS 2021.
No
We would like to thank the reviewer for the insightful and positive comments.
Intuitively, hypercontractivity states that for any policy, features of the induced state-action distribution satisfy certain anti-concentration properties. Since Gaussian distributions (with arbitrary covariance matrices) satisfy hypercontractivity, a practical example where hypercontractivity holds is the case when the state transition has Gaussian noise (as common in control settings). We will add more discussion in the next version.
LOWER BOUND:
The construction for the bound seems to be largely inspired by Weisz et al. 2020’s construction: the crucial difference to achieve the suboptimality gap is to construct a "leaky” graph instead of downscaling the features. This way, the downscaling is essentially achieved by the transitions. The two techniques allow for a similar information theoretic argument to finish the bound. The result is both stronger and weaker: the suboptimality gap is a stronger assumption, but the lower bound only works in the online RL setting (as opposed to the generative model).
I was happy to see a hard MDP construction similar to Weisz et al 2020’s presented (in my opinion) much more cleanly.
- I am missing the definition of Pr_M at appendix line 61.
- I am unsure why the appendix "addressing footnote 3" is necessary: could we not just bijectively remap the m-1 actions of each state to [m-1]?
UPPER BOUND:
I would like to see some assumptions clarified here, as there might be inaccuracies. Below is my best attempt at understanding them. Whether I am right or wrong here, the effort required to address my issues in the paper should be fairly small.
- First, in line 147, I believe the precise assumption on theta* and phi's 2-norms is that they are <=1 (instead of O(1); see eg appendix line 189). I would like to see this stated as a separate assumption. The trouble with the argument that this "can always be achieved via rescaling" is that one would have to rescale the rewards, and consequently epsilon too, to achieve the same guarantee. This means that the final bound will scale polynomially with these 2-norms, instead of (what might be possible) logarithmically. I believe for this reason the assumption should be made explicit, as it is in Du et al 2019b. It would be nice for this assumption to be referenced in the proof too, (eg appendix lines 103, 189, lemma 7, etc.).
- Second, in lemma 7, Du et al 2019b’s relevant lemma is A.3 instead of A.2, (at least in the versions available to me), and it is stated slightly differently. I would like to see these differences proved or explained. These are:
- a condition on eta<=1
- |\xi|<=1 (I'm not sure what n refers to in appendix line 149)
- eta^2 in the bound instead of eta
- Third, Du et al 2019b uses an additional assumption, that rewards are nonnegative and the sum of rewards on *any* trajectory is bounded by 1. I believe this assumption is missing from this paper, but seems to be implicitly used at places like (other than possibly at lemma 7): appendix line 98 (for the value bound), and line 107 (to bound \xi almost surely; here this bound would be <=1, not sure where the 2 comes from). Both of these arguments feel somewhat unfinished to me.
For the low variance and hypercontractivity assumptions introduced at the beginning of section 5, it would be nice to see examples or descriptions of the kinds of MDPs whose value functions satisfy this assumption. The current description focuses on the value functions, not on the MDPs that might induce such value functions. Having this could help interpreting the generality of these assumptions (and hence the corresponding upper bound too).
Does the upper bound work under a near-realizability assumption (eg. ||Q*-phi^T theta*||_\infty <= something *very* small)?
- Line 315: is known -> was known
- Line 7: such a lower bound (add bound)
- Appendix line 119: consider better justifying line 3 to line 4, bound of theta-theta* by including the calculation
No
We would like to thank the reviewer for carefully reading our submission and providing many insightful comments.
1. Here we used $\Pr_M[s' | s, a]$ to denote the probability of transiting from $(s, a)$ to $s'$ in MDP $M$. We will change the notation to $P_M(s' \mid s, a)$ in the next version to make the notation consistent.
2.
Indeed remapping the actions will result in a much simpler proof. We will adopt such proof in the next version. Thanks for the suggestion.
3. We agree that the precise assumption on theta* and phi's 2-norms is that they are <=1, and that will be made clear in the next version.
4. The condition that $\eta \le 1$ was missed in the statement of Lemma 7 (note that the error bound was correctly stated since in our case $\eta = \overline{b}^2$). The correct constraint on $\xi$ should be $|\xi| \le 2$. Note that this will only increase the sample complexity by a constant factor which was hidden in the $\Omega$ notation. These typos will be fixed in the next version.
5. We did assume that rewards are nonnegative and the sum of rewards on any trajectory is bounded by 1, and that will be made clear in the next version. We will also connect relevant parts in the proof to this assumption for clarity.
We note that hypercontractivity is an assumption on the distribution of the features instead of the value functions. Intuitively, hypercontractivity states that for any policy, the features of the induced state-action distribution satisfy certain anti-concentration properties. Since Gaussian distributions (with arbitrary covariance matrices) satisfy hypercontractivity, a practical example where hypercontractivity holds is the case when the state transition has Gaussian noise (as common in control settings). We will add more discussion in the next version.
For the agnostic setting where realizability holds approximately, we believe a similar proof will work.
However, in this case, the required agnostic error will depend on the gap. See [1] for more discussion on the agnostic setting.
[1] Agnostic q-learning with function approximation in deterministic systems: Near-optimal bounds on approximation error and sample complexity.
I would like to thank the authors for addressing my questions and concerns. With this in mind, I have changed my rating from 7 to 8.

Meta-review:
role: PC
data: comment: The reviewers all liked the paper very much. It will be a very nice addition to the conference program, presenting interesting and important additions to the growing body of literature on the theoretical analysis of RL algorithms with function approximation under the assumption that the action-value function of the optimal policy ($Q^*$) is linearly realizable.

Decision: Accept (Oral)