Title: Online RL in Linearly $q^\pi$-Realizable MDPs Is as Easy as in Linear MDPs If You Learn What to Ignore
Venue: NeurIPS 2023
Accepted: yes
Keywords: Reinforcement learning

linear function approximation

online learning
Abstract: We consider online reinforcement learning (RL) in episodic Markov decision processes (MDPs) under the linear $q^\pi$-realizability assumption, where it is assumed that the action-values of all policies can be  expressed as linear functions of state-action features. This class is known to be more general than  linear MDPs, where the transition kernel and the reward function are assumed to be linear functions of the feature vectors. As our first contribution, we show that the difference between the two classes is the presence of states in linearly $q^\pi$-realizable MDPs where for any policy, all the actions have  approximately equal values, and skipping over these states by following an arbitrarily fixed policy in those states transforms the problem to a linear MDP. Based on this observation, we derive a novel (computationally inefficient) learning algorithm for linearly $q^\pi$-realizable MDPs that simultaneously learns what states should be skipped over and runs another learning algorithm on the linear MDP hidden in the problem. The method returns an $\epsilon$-optimal policy after $\text{polylog}(H, d)/\epsilon^2$ interactions with the MDP, where $H$ is the time horizon and $d$ is the dimension of the feature vectors, giving the first polynomial-sample-complexity online RL algorithm for this setting. The results are proved for the misspecified case, where the sample complexity is shown to degrade gracefully with the misspecification error.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis submission provides the first algorithm learning an $\\epsilon$-optimal policy with poly many samples obtained through online interactions with a $q^\\pi$-realizable MDP. This is an important contribution to an open question in the community. Indeed previous works either adopted a generative model or they required more stringent condition on the MDP class such as Linear MDP.\nThe core observations that might be useful for futures work are that:\n1) $q^\\pi$-realizability + strictly positive lower bound on the range $\\implies$ Linear MDP.\n2) In a $q^\\pi$-realizable MDP not only $q^\\pi$ are in the features span but also all admissible (see definition 4.6) functions.\n\nUnfortunately, the proposed algorithm is computationally inefficient as the authors mention at line 35 in the introduction.\nI think that this limitation should have been better discussed in the paper and mentioned in the abstract as well.\n\nSTRENGTHS:\nLearning an $\\epsilon$-optimal policy with poly many samples obtained through online interactions with a $q^\\pi$-realizable MDP was an open question before this paper (to the best of my knowledge). This paper solves it with an elegant approach unveiling the fact that $q^\\pi$-realizable MDP are not linear MDP due to the existence of some low range states.\nHowever this states are unimportant for the goal of finding an optimal policy and therefore in these states we can fix an arbitrary policy and reduce a $q^\\pi$-realizable MDP to a Linear MDP with features norm bound inversely proportional to the range lower bound.\nThe authors further develop their algorithm to bypass the requirement of knowing the low range states in advance.\nI think this reduction to Linear MDP will be important for further advancements in the field.\n\nWEAKNESSES:\nI think the authors should have discussed better the fact that their algorithm is not computationally efficient.\nIn particular, I believe that the authors should explain that the computational barrier comes from Optimization Problem 4.10. This is probably obvious for reader already familiar with ELEANOR but I would still suggest to include this discussion in a Limitations section that the authors may want to add in their final revision. \n\nSome steps in the presentation might also be improved (please see questions in the next Section)\n\nQUESTIONS:\n1) Which is the output of Algorithm 2 ? In Algorithm 1, at line 7 the policy $\\pi^{mk}$ is defined as output of Skippy Policy which is not specified. Is it the policy that at state $S_i$ sample an action from $\\pi^+(S_i)$ with probability $\\tau_i$ and plays action 1 otherwise ?\n\n2) Is the quantity $E^{\\rightarrow}(s_i \\rightarrow)$ at line 275 be the same as the quantity $E_{G\\theta}^{\\rightarrow}(S_{t+1}^{lkj}, \\dots R_H^{lkj})$ used in the least squares target ?\n\n3) I couldn't follow your reasoning to show that $E_{G\\theta}^{\\rightarrow}(S_{t+1}^{lkj}, \\dots R_H^{lkj})$\n  is linearly realizable. In particular, Corollary 4.11 shows that is realizable for the true guess. However the solution to the Optimization Problem 4.10 is different from the true guess so in this case $E_{G\\theta}^{\\rightarrow}(S_{t+1}^{lkj}, \\dots R_H^{lkj})$ can not be guaranteed to be realizable. Could you please elaborate more on this step in the rebuttal ?\n\n4) Given the reduction from $q^\\pi$-realizable MDPs to Linear MDPs that you discovered in this paper. What would prevent one from applying efficient algorithms for linear MDPs such as LSVI-UCB to solve $q^\\pi$-realizable  MDPs ? Why did you choose to adopt the computationally inefficient Eleanor as base algorithm for solving Linear MDPs while computationally efficient alternatives exist ?\n\nLIMITATIONS:\nThe limitations are not well discussed. In particular, I would suggest to add a discussion concerning the computational cost of the algorithm.\n\n**Minor**: typo at line 302 \"th matrix\" -> \"the matrix\"", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proves that the standard online episodic RL with the $q^\\pi$-realizability assumption can be learned with polynomial samples, without generative models. Technically, this paper proves that the states of any MDPs with $q^\\pi$-realizability assumption can be partitioned into two disjoint sets, where the states in the first set have small range (i.e., the advantage function on the state is always small for every policy), and the transitions on the states in the second set can be approximated by a linear MDP. Consequently, this paper designs a novel algorithm called SKIPPYELEANOR that learns to distinguish the states with small range and the linear MDPs simultaneously.\n\nSTRENGTHS:\nThis paper solves a long-standing open question of reinforcement learning theory by showing that MDPs with $q^\\pi$-realizability assumption can be learned with polynomial samples. The result is significant and beneficial to the community. The observation that the difference between MDPs with $q^\\pi$-realizability and linear MDPs is only the set of states with very small range is novel and neat. Conceptually, this observation deepens our understand on the $q^\\pi$-realizability assumption and can potentially inspire a new line of research.\n\nWEAKNESSES:\nThe exposition of the results could be improved. Current Section 4 of this paper is very technical and involves too many symbols and definitions, which makes it less readable, despite the fact that the algorithm is conceptually simple. Is there a simpler setting that demonstrate the core idea of the algorithm while keeps the technical part less involved?\n\nQUESTIONS:\nIn the last paragraph in Section 3, this paper argues that a MDP with $q^\\pi$-realizability can be converted to linear MDP by skipping the states with low range. Is this a rigorous reduction? How to construct the new MDP? In particular, the states are staged in the original MDP. By skipping some states, is the new MDP also staged, and is the stage of the un-skipped states unchanged?\n\n[Minor] In the caption of Figure 1, should the feature of all other states be 0 instead of 0.5?\n\n[Minor] What’s $S_h$ in the displayed equation after line 222?\n\nLIMITATIONS:\nThe authors adequately addressed the limitations and potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies online RL under linear $q^\\pi$ realizable MDPs, and proposes an algorithm that learns an epsilon-optimal policy in polynomial time and polynomial sample complexity.  Previous online RL in linear RL either assumes that transitions and rewards are linear individually (e.g., linear MDP), or assumes ergodicity (e.g., Politex), or assumes access to an simulator that can reset to a previously visited states (Confidence-MC-Politex). This work removes all these assumptions and show that as long as $q^\\pi$ is realizable for all $\\pi$, polynomial-time algorithm is possible. The novel and surprising observation is that linear $q^\\pi$ MDP is actually a linear MDP if the \"range\" of all states are large. The proposed algorithm is inspired by ELEANOR but \"skip\" low-ranged state in its execution, and at the same time refine the estimation for the range on the fly using some techniques related to optimal design.\n\nSTRENGTHS:\nThe observations and techniques in this paper are novel and creative. It answers an open question by Du et al. (2019) on whether sample-efficient RL under linear $q^\\pi$ MDP is possible. \n\n[Du et al., 2019] Simon Du, Sham Kakade, Ruosong Wang, Lin F. Yang. Is a Good Representation Sufficient for Sample Efficient Reinforcement Learning?\n\nWEAKNESSES:\nWhile I appreciate the technical contribution, the delivery of the result is far from ideal. The paper is very difficult to read. I have identified the following possible reasons: \n1. Highly cluttered notation.  I think a rule of thumb in writing math is to avoid super/subscript of super/subscript as best as we can, and keeping the super/subscript as simple as possible, because they can make the expression difficult to parse. However, this kind of notations is everywhere in this paper, like $\\hat{\\theta}\\_{\\hat{G}\\bar{\\theta}}^{p^{mkj}(k),i}$ or \n$||\\varphi_{p(k)}^{mkj}||\\_{X^{-1}_{m,p^{mkj}(k)}}$ where not only the super/subscript dependence is deep, each layer has many letters. It becomes a significant obstacle when tracing the calculations. Besides, for the notation $p^{mkj}(k)$, I did not find a place where the parameter in the parenthesis does not match the second parameter in the superscript, so maybe one of them can be omitted? \n2. The \"definition\" dependence is deep. This means that the definition of a quantity usually involves another quantity defined somewhere else; and when I visit the \"somewhere else\", it further requires me to understand other quantities defined in other places, and so on. For example, Optimization Problem 4.12 involves $y^{ki}\\_{\\hat{G}\\bar{\\theta}}$ and \n$\\hat{F}^{ki}\\_{\\hat{G}\\bar{\\theta}}$ for which I have to check the complicated Eq. (12) for their definitions. The $c^j_{ki}$ \nthere is further defined by a big quantity in (18) and the definitions of $\\varphi^{mkj}\\_{p(k)}$ and $\\hat{\\theta}\\_{\\hat{G}\\bar{\\theta}}^{p^{mkj}(k), i}$ are further scattered in the previous page which further requires understanding other quantities first. I think part the difficulty to trace is also because of Reason 1 --- the readers are facing highly cluttered notations in each step of tracing the definitions, and eventually cannot really grasp the overall meaning.  \n3. No reminders for the place of the definitions.  Since there are a big amount of cluttered notations, the readers easily forget where they are defined, so I think it's better that there could be reminders here and there to let the reader know where the notations are defined previously.  For example, when I first see $C_{\\hat{G}\\bar{\\theta}}$ in Line 279, I don't know where this is defined. After a long time, I realized that it is a result of the statement in Line 256-258 with the $C$ defined in (8). \n4. Confusing notation/terminology: In Line 188, $Z(Q,h)$ is defined as some subspace, but $Proj_{Z(Q,h)}$ is defined as projection onto its *orthogonal* space, which is counterintuitive notational-wise. As another example, in Line 266 or Line 4 of Algorithm 2, $\\tau$ is called a \"skipping probability\", but to my understanding, $\\tau$ is more like \"probability of *not* skipping\". \n5. Lack of high-level explanation. Though I can see that the authors have used a considerable space for describing the components of the algorithm, there are still many design choices left unexplained. For example, I don't quite understand why an additional index of k in Line 6 of Algorithm 1 is needed, and why we need to call Algorithm 2 with that input k. I also hope to see a more high-level interpretation of the $\\mathbf{f}$ defined in Line 240, or what does it implied if $v^\\top_{\\||(Q,h)} \\mathbf{f}w$ is $\\alpha$-admissible (Lemma 4.9 and Corollary 4.11). For example, what's the importance of this fact for the checking consistency procedure. The design of the regression target $E^{\\rightarrow}(s_i\\rightarrow)$ also deserves more explanation, e.g., what's the relation between this and the standard one in ELEANOR. \n\nI understand that the main text space is very limited. But I believe there is way to give a high-level explanation and self-contained story in the main texts, and leaving details to the appendix. Currently, the main text itself is crowded with difficult-to-trace notation and lacks necessary explanations. This also hinders me from making a careful check for the correctness of the proofs. So I suggest that the authors spend time re-thinking about how to present the results, to really make it a strong paper. \n\nOther notations I couldn't find corresponding definitions: \n- $B_i$ in Line 5 of Algorithm 2\n- $\\otimes$ in (12).\n\nQUESTIONS:\nCan you give some high-level ideas of Optimization Problem 4.12? What are the meanings of $y^{ki}\\_{\\hat{G}\\bar{\\theta}}$ and $\\hat{F}^{ki}\\_{\\hat{G}\\bar{\\theta}}$? \n\nBesides, why do we need the for-loop in Line 6 in Algorithm 1?\n\nLIMITATIONS:\nThere's no potential societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses online RL in episodic MDPs under linear $q^\\pi$-realizability, where policy action values are expressed as linear functions of state-action features. This class is shown to be more general than linear MDPs, where only the transition kernel and reward function are linear functions of feature vectors. The key contribution is identifying states in linearly $q^\\pi$-realizable MDPs where actions have approximately equal values, enabling transformation into a linear MDP. A novel learning algorithm is derived to determine which states to skip and apply another learning algorithm to the linear MDP. The proposed algorithm achieves an $\\varepsilon$-optimal policy after $\\operatorname{polylog}(H, d) / \\varepsilon^2$ interactions, representing the first online RL algorithm for linearly $q^\\pi$-realizable MDPs with polynomial sample complexity. Notably, the results hold in the misspecified case, where sample complexity gracefully degrades with the misspecification error.\n\nSTRENGTHS:\n1. This paper offers a comprehensive exploration of Linearly $q^{\\pi}$-Realizable MDPs, making a significant contribution to the field of reinforcement learning with feature learning.\n\n2. The paper exhibits a clear and concise mathematical exposition with a well-structured presentation that is easy to follow.\n\nWEAKNESSES:\n1. The current form of the paper lacks a Conclusion section, rendering it incomplete.\n\n2. It would enhance clarity if the authors presented a comparative analysis in tabular form, providing a clearer understanding of the different studies involving MDPs with $q^{\\pi}$-realizability and linear structure.\n\n3. The paper has relatively few references, and the discussion of this line of work appears somewhat limited. It would be beneficial for the authors to discuss more related work in this area.\n\n4. The left and right sides of Figure 1 seem to be reversed.\n\nQUESTIONS:\n1. What is the lower bound for MDPs with $q^{\\pi}$-realizability? It appears to have significant dependencies on both the time H and d.\n\n2. There seems to be an issue with the citation format. The requirement is to use an unnumbered first-level heading for the references.\n\nLIMITATIONS:\nThe suggestions have been claimed in \"Weaknesses\" and \"Questions\".", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}