Title: Computational Hardness of Reinforcement Learning with Partial $q^{\pi}$-Realizability
Venue: NeurIPS 2025
Accepted: yes
Keywords: Reinforcement Learning

Linear Function Approximation

Computational Complexity

NP-Hardness
Abstract: This paper investigates the computational complexity of reinforcement learning within a novel linear function approximation regime, termed partial $q^{\pi}$-realizability. In this framework, the objective is to learn an $\epsilon$-optimal policy  with respect to a predefined policy set $\Pi$, under the assumption that all value functions corresponding to policies in $\Pi$ are linearly realizable. This framework adopts assumptions that are weaker than those in the $q^{\pi}$-realizability setting yet stronger than those in the q*-realizability setup. As a result, it provides a more practical model for reinforcement learning scenarios where function approximation naturally arise. We prove that learning an $\epsilon$-optimal policy in this newly defined setting is computationally hard. 
More specifically, we establish NP-hardness under a parameterized greedy policy set (i.e., argmax) and, further, show that—unless NP = RP—an exponential lower bound (exponential in feature vector dimension) holds when the policy set contains softmax policies, under the Randomized Exponential Time Hypothesis. Our hardness results mirror those obtained in the $q^*$-realizability settings, and suggest that computational difficulty persists even when the policy class $ \Pi $ is expanded beyond the optimal policy, reinforcing the unbreakable nature of the computational hardness result regarding partial $ q^{\pi} $-realizability under two important policy sets. To establish our negative result, our primary technical contribution is a reduction from two complexity problems, $\delta$-Max-3SAT and $\delta$-Max-3SAT($b$), to instances of our problem settings: GLinear-$\kappa$-RL (under the greedy policy set) and SLinear-$\kappa$-RL (under the softmax policy set), respectively. Our findings indicate that positive computational results are generally unattainable in the context of partial $ q^{\pi} $-realizability, in sharp contrast to the $ q^{\pi} $-realizability setting under a generative access model.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the computational complexity of reinforcement learning algorithms under the partial q^pi-realizability setting, a novel framework introduced by the authors.\n\nThey first establish NP-hardness when considering a class of greedy policies. Furthermore, under the Randomized Exponential Time Hypothesis, they show that learning with softmax policy classes has exponential computational complexity in the dimension of the feature vector.\n\nThe proposed setting and results help bridge the gap between the linear q* and linear q^pi assumptions, offering a more complete understanding of the landscape.\n\nA key technical contribution is the reduction from two well-known complexity problems:  delta-MAX-3SAT and delta-MAX-3SAT(b).\n\nSTRENGTHS AND WEAKNESSES:\nThis paper is well motivated, as it explores the intermediate regime between two fundamental settings in reinforcement learning theory: linear q* and linear q^pi.\n\nThe discussion of related work is thorough and informative, providing valuable context for the reader.\n\nThe assumption that the agent has access to a generative model strengthens the significance of the hardness results presented.\n\nPrevious work has shown quasi-polynomial and even exponential lower bounds under the q*- and v* -realizability assumptions. On the other hand, under the stronger assumption of linear q^pi-realizability, efficient algorithms exist when using a generative model. This naturally raises the question: Can the positive results be generalized, or do hardness results still hold under more relaxed assumptions? \n\nThis paper provides a compelling answer, demonstrating that computational hardness persists even when the policy class extends beyond the optimal policy pi*, that is, beyond the linear q* setting.\n\nThe two main results (Theorems 3.1 and 3.2) are both significant. The core technical contribution lies in clever reductions from classical complexity problems, which underpin the hardness proofs.\n\nAlthough the proposed partial linear q^pi setting is novel, it appears to lack natural motivating examples.\n\nQUESTIONS:\nThe proposed partial q^pi setting is quite interesting. Could you provide some natural or practical examples that illustrate this setting?\n\nThe paper mentions that related works use reductions to establish hardness results. Could you elaborate on how the reduction used in this paper differs from those in prior work?\n\nLIMITATIONS:\nnot applicable", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new intermediate computational problem called _partial $q^{\\pi}$-realizability_, along with new complexity analysis for this setting. Prior work has examined two end points related to this new setting termed $q*$-realizability (in which $q^*$ is representable using a linear combination invoking features, $\\phi$) and $q^\\pi$-realizability (in which $q^\\pi$ is representable for all policies). Here, the problem of partial $q^\\pi$-realizability_ is defined _relative_ to a choice of policy class, $\\Pi$, which allows the problem to soften its requirements: only the $q$ functions of the policies contained in the set must be representable. Then, the second contribution of the work is new analysis of this partial realizability problem. The analysis comes in two Theorems which differ in which policy class the result centers around. The first (Theorem 3.1), makes use of the class of deterministic greed policies, $\\Pi^g$, while the second (Theorem 3.2) uses softmax policies; in both cases, the corresponding versions of the problem are proven to be hard. The rest of the paper is dedicated to an overview of the proof technique, involving details about the reductions used and examples.\n\nSTRENGTHS AND WEAKNESSES:\n__Strengths__\n- [New, Interesting Problem] The relaxation of realizability to the partial case is a clever idea, and presents an interesting new problem.\n- [Fundamental Result] The new hardness results are fundamental and interesting.\n- [Focused Narrative] This paper is ultimately about one clean idea: partial $q^\\pi$ realizability\n- [Clarity] Despite the technical nature of the paper, it is very readable. A lot of work clearly went into making the ideas accessible.\n\n__Weaknesses__\n- There are no major weaknesses. For what this paper sets out to do, it executes on it to a high degree.\n- Two minor weaknesses: (1) I could imagine clarifying some of the design choices in slightly more detail. For example, why did you choose the softmax policies to analyze, rather than another class? Do you anticipate similar results hold for most well-behaved choices of policy class? (2) The different feature vectors did leave me slightly confused by the end. Adding a little more exposition to help the reader understand this will help in communicating some of the main ideas.\n\n\nIn light of the above, unless another reviewer catches a glaring issue, I believe this is a clear accept.\n\nQUESTIONS:\nI only have a few high-level suggestions:\n- You might consider restructuring Sections 3.1 and 3.2. As a reader, my expectation after having the gLINEAR-$\\kappa-RL problem defined was then to dive into the theorem and analysis around this problem. The flow is slightly disjointed by moving between the two problems, then going onto the analysis. But, I recognize this is personal taste, and I see the tidiness of the current framing as well. Just something to consider.\n- For Theorem 3.1, the proof works for \\epsilon_0 = 0.05. For this reason, I would state the theorem using 0.05 directly and not bother introducing epsilon_0.\n- I can imagine it would be useful to motivate the choice of softmax policies a bit further. Or, at least to comment on whether you believe Theorem 3.2 is tailored to the choice of $\\Pi^{sm}$, and to what extent you believe a similar/nearly-identical result will hold for most well-behaved policy classes.\n- As you'll see, the abstract does not render well on openreview.\n\n__Typos / Writing Suggestions__\n\nThen, I include a few low-level reactions and suggestions below.\n\nAbstract:\n- You can safely remove this \"(a variant of MAX-3SAT)\", as the names of the problems makes this clear already.\n\nIntroduction:\n- Supporting some of these statements with references would be useful: \"...the estimation of the value function so that resulting complexity bounds become independent of the size of the state space\"\n- For Q2, the symbol \"$\\subsetneq$\" seems unecessary here, rather than just $\\subset$, right? Oh, I see, you want to specify that $\\pi^*$ is in the set, but not the only policy in the set?\n- Line 69: \"by defining subclass\" --> \"by defining the subclass\"\n- Line 70: \"of realizable policy set\" --> \"of realizable policy sets\"\n- Equation between lines 106/107: You use $\\sup$ in all cases one but, where you move to the $\\max$; any specific reason?\n\nPreliminaries\n- Line 114 header: \"Linear Function Approximations\" --> \"Linear Function Approximation\"\n\nProblem Statement and Main Results\n- Definition 3.3: I assume there is a good reason to make use of the states reachable after $h$ steps: $\\mathcal{S}_h$, though it seems to add complexity for little benefit so far. \n- Since the softmax policies do not use a temperature parameter, you might flag this somewhere early on. For instance, in the introduction, when you mention you will study softmax policies, you could add a short note indicating that there is no temperature used.\n- Typo: \"Theorem 3.1 (NP-Harness\" --> \"Theorem 3.1 (NP-Hardness\"\n- For Theorem 3.1: Is your intuition that the result also holds for values larger than 0.05, but that your proof simply won't generalize to those cases? Or, do you suspect for some larger epsilon that the problem becomes easier (this would be surprising)?\n\nLIMITATIONS:\nYes, the authors call out all assumptions and limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nPrior work has shown that RL is computationally tractable under the stronger $q^\\pi$-realizability assumption, whereas the weaker $q^*$-realizability assumption leads to exponential lower bounds. To investigate whether improved results are achievable under assumptions weaker than $q^\\pi$-realizability, this paper introduces a partial realizability assumption. It then considers two types of policy classes—greedy and softmax—and proves an exponential lower bound on the computational complexity in both cases.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThe paper is well written, providing a clear introduction that conveys the intuition behind the problem and offering a detailed overview of the proof strategy. Although I did not verify every step of the proof, the theoretical analysis appears to be solid. The paper presents a novel result on the computational complexity of the RL problem, which, to the best of my knowledge, has not been previously reported in the literature.\n\n---\n\nWeaknesses:\n\n(1) Definition 3.1 is somewhat ambiguous, as the weight parameter $\\theta$ should include a superscript $\\pi$ to indicate its dependence on the policy.\n\n(2) One concern I have is that the result is entirely negative and does not hold for every policy class. In particular, because the feature map $\\phi'$ is constructed as part of the proof, the result does not offer insight into which structural properties of a policy class might help reduce the problem’s complexity. Is there any example of a policy class with alternative feature representations that could potentially avoid this hardness? Furthermore, given that the computational hardness of RL under $Q^*$-realizability is already well-established, it is unclear to me what new understanding or contribution this paper adds relative to the existing body of work.\n\n(3) I find the paper to be closely related to [1], especially in its approach to constructing an MDP from a SAT instance. However, it lacks a comprehensive comparison of its proof techniques with those of [1] and other relevant prior works.\n\n[1] Exponential Hardness of Reinforcement Learning with Linear Function Approximation, Kane et al. 2023\n\nQUESTIONS:\n(1) The claims in Lines 214–219 seem a bit unclear to me. Specifically, the policies in $\\Pi_g$ are deterministic, while those in $\\Pi_{sm}$ are stochastic. Why should $\\Pi_g \\subseteq \\Pi_{sm}$ hold with high probability? Also, over what is this probability defined? \n\n(2) Theorem 3.2 does not explicitly claim that SLINEAR-κ-RL is NP-hard. This should perhaps be stated directly in the theorem.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the computational complexity of reinforcement learning under linear function approximation. The authors introduce a new notion, partial $q^{\\pi}$-realizability, which lies between the existing assumptions of $q^*$-realizability and $q^{\\pi}$-realizability. They show that finding an $\\varepsilon$-optimal policy within the greedy policy class is NP-hard, via a polynomial-time reduction from $\\delta$-MAX-3SAT. For the softmax policy class, they establish an exponential time lower bound in the feature dimension under the randomized Exponential Time Hypothesis (rETH). The authors conclude that even when the policy class is relaxed beyond the optimal policy, the computational hardness of reinforcement learning remains unchanged.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nThe paper proposes a novel realizability assumption called *Partial* $q^\\pi$-realizability, which bridges the gap between the existing $q^*$- and $q^\\pi$-realizability assumptions. It establishes fundamental computational hardness results under linear function approximation by leveraging different complexity hypotheses for two distinct policy classes. The logical structure of the paper is clearly organized, making it easy to follow.\n\n\n**Weaknesses**\n\n\nAs acknowledged in the paper’s limitations, it remains unaddressed whether similar lower bounds hold in the case of shared features.\n\nQUESTIONS:\n1. Under the *partial* $q^\\pi$-realizability assumption, what additional conditions would allow for efficient learning?\n2. Although the paper is theoretical in nature, it would be more convincing if a small illustrative MDP example demonstrated the computational failure or blow-up of existing linear function approximation algorithms. Is there a simple experiment that could be conducted?\n3. Could similar lower bounds be shown under weaker access assumptions than the generative model? If not, could you explain what technical challenges prevent this?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 1, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}