Title: Near-Optimal Sample Complexity for Online Constrained MDPs
Venue: NeurIPS 2025
Accepted: yes
Keywords: Reinforcement learning

constrained MDPs

online learning

sample complexity

learning theory
Abstract: Safety is a fundamental challenge in reinforcement learning (RL), particularly in real-world applications such as autonomous driving, robotics, and healthcare. To address this, Constrained Markov Decision Processes (CMDPs) are commonly used to enforce safety constraints while optimizing performance. However, existing methods often suffer from significant safety violations or require a high sample complexity to generate near-optimal policies. We address two settings: relaxed feasibility, where small violations are allowed, and strict feasibility, where no violation is allowed. We propose a model-based primal-dual algorithm that balances regret and bounded constraint violations, drawing on techniques from online RL and constrained optimization. For relaxed feasibility, we prove that our algorithm returns an $\varepsilon$-optimal policy with $\varepsilon$-bounded violation with arbitrarily high probability, requiring $\tilde{O}\left(\frac{SAH^3}{\varepsilon^2}\right)$ learning episodes, matching the lower bound for unconstrained MDPs. For strict feasibility, we prove that our algorithm returns an $\varepsilon$-optimal policy with zero violation with arbitrarily high probability, requiring $\tilde{O}\left(\frac{SAH^5}{\varepsilon^2\zeta^2}\right)$ learning episodes, where $\zeta$ is the problem-dependent Slater constant characterizing the size of the feasible region. This result matches the lower bound for learning CMDPs with access to a generative model. Episodic tabular CMDPs serve as a crucial benchmark for safe RL, providing a structured environment for theoretical analysis and algorithmic validation. Our results demonstrate that learning CMDPs in an online setting is as easy as learning with a generative model and is no more challenging than learning unconstrained MDPs when small violations are allowed.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies tabular CMDPs. They proposed a primal-dual algorithm that obtains a near-optimal policy with sample complexity that matches the lower bound. This holds for both the setting considered in the paper, relaxed and strict feasibility.\nThe paper is generally well-written and easy to follow. I had several questions while reading—some were answered later, but it would still be helpful to clarify them in the final version for the benefit of other readers.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\nThe theoretical results are solid. They provide upper bounds for their algorithm, which match the lower bounds existing in the literature.\n\nWeakness: There were no major flaws/weaknesses in the paper. \n- The assumption of a finite state-action space is somewhat limiting in practice. The model essentially reduces to a linear setting. (This is only my opinion about the considered settings in the literature; however, there are multiple works with similar setting,s and this paper adds value)\n- The paper would benefit from an implementation of the algorithm and releasing the code.\n- It would be helpful to clearly state all assumptions using an Assumption environment and refer to them explicitly in the theorems. (But not sure if there are any assumptions apart from written in the problem setup) For example, do you require full support over the state space from the initial state distribution?\n- See the questions below for improving paper clarity\n\nQUESTIONS:\nWhat does giving access to generative model means? Does it imply that system can obtain transitions values for any state, looks impractical?\n\nline 71-71: It was counterintuitive why sample complexity depends inversely on the size of the feasible region.\n\nline 162: explain intutively \\tilde{r}. Is it some optimistic acquisition function?\n\nIn section 3: problem setup can you write the finite state space and finite action spaces? It is mentioned in line 36 but no where in problem setup section.\n\nMethodology section starts nicely with explaining why the existing methods have loose bounds. Since the section is long with a sequence of paragraphs, i recommend naming paragraphs as \"\\textbf{name.} content of paragraph.....\"\n\nline 174-181: Can you also clarify why the doubling batch updates solves the problem\n\nIt would be good to mention t, k, and h early on and what they represent. In line 202 why does a finite number of solve iterations T is Guaranteed to converge to optimal solution?\n\nline 233, why \\pi^{\\tu, \\star} is a deterministic policy?\n\nin the proof of theorem 1, why does s_1 depends on k? is it a typo? If it does depend on k, then \\pi^{\\tau, \\star} should also depend on k.\n\nline 237-238: Is this term (3rd regret term) non-zero due to finite number of gradient steps T? If it T is long enough (converges) then it will be zero as well? I think so, but it is not evident with the upper bounds of this term. Is your regret optimal with T?\n\nWhat does _{+} represents is CV(K)? If it is ciel function, shouldn't it be inside \\sum_k ? \n\nTheorem 4: why there is constraint violations in the case of strict feasibility? (c.f. line 14 saying zero violation)\n\ntheorem 3: does \\delta' h in line line 264 is same as line 184? Since theorem statement is with 1-\\delta, may be just write 1-SHAK\\delta' > 1-\\delta.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the online learning problem for episodic constrained MDPS.\nFor the proposed online learning algorithm, which does not require access to a generative model(as in [3]), they provide regret bounds, constraint violation bounds, and sample complexities for relaxed and strict constrained MDP scenarios.  The algorithm and proofs leverage many existing ideas from constrained MDPs and unconstrained MDPs to improve the regret bounds compared to other existing algorithms.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths** \n\n- The paper addresses many theoritical aspects of the constrained MDP learning problems by providing regret bounds, constraint violation bounds and sample complexity in both relaxed and strict feasibility cases. \n- The paper removes the requirement of access to generative model assumed in [3]. \n- The presentation and writing in this paper is good and easy to follow, particularly the inclusion of Section 4 provides lot of insights into the existing work and the algorithm.\n\n\n**Weaknesses**\n- The paper mentions that constrained MDP is an important setup for RL problems with safety concerns. The paper's contribution is purely algorithmic and theoritical. It lacks any experimental / simulation studies, comparison with existing algorithms. As seen in unconstrained RL algorithms, particularly UCRL variants, algorithm with similar regret bounds have been shown to have different empirical performance, so improved regret bounds alone are not sufficient to make a strong case for a new algorithm. \n\n- Please see the questions section for potential weaknesses and questions. \n\n**References**\n\n[1]  Tao Liu, Ruida Zhou, Dileep Kalathil, P. R. Kumar, and Chao Tian. Learning policies with zero\nor bounded constraint violation for constrained mdps. In Proceedings of the 35th International\n Conference on Neural Information Processing Systems, NIPS ’21, 2021.\n\n[2] Gen Li, Laixi Shi, Yuxin Chen, Yuantao Gu, and Yuejie Chi. Breaking the sample complexity\n barrier to regret-optimal model-free reinforcement learning. In Proceedings of the 35th International Conference on Neural Information Processing Systems, NIPS ’21, Red Hook, NY,\n USA, 2021.\n\n[3] Sharan Vaswani, Lin Yang, and Csaba Szepesvári. Near-optimal sample complexity bounds\nfor constrained MDPs. In Advances in Neural Information Processing Systems, 2022\n\nQUESTIONS:\n**Technical Questions**\n\n- The algorithm uses optimistic rewards and cost functions along with the empirical model, even though the reward and cost functions are known. In unconstrained RL (UCRL2 variants ) as well as in constrained RL [1], optimistic estimates are used for the unknown quantities ($P$ in the case). Can the author provide their intuition on why they do not use optimistic estimates of P instead of the MLE estimate, as well as use optimistic reward and cost when they are assumed to be known?  When optimism is required and when empirical estimate suffices? This is similar to [3] but more surprising as they assume access to a generative model.\n\n\n- As mentioned earlier, even though the algorithm is shown to have improved pre-constants in regret, it does not give a complete picture of the comparison of various model-based constrained RL algorithms.  A common theme in many unconstrained, tabular RL papers is to provide empirical comparison with existing algorithms in terms of regret and computational complexity; this has been missing in most Tabular constrained MDP papers, including this one. Can authors potentially address this missing aspect in this line of papers, which can provide new insights into these algorithms?\n\n- Why is $\\pi^{\\tau,*}$ a deterministic policy (Line 233)?\n- As suggested in Lines 55-56 (\"large-scale state spaces\"), what aspect of the algorithm makes it more suitable for large state spaces compared to the previous algorithms? \n\n \n**Writing / Presentation Questions (and Suggestions)**\n- As found in many theoretical RL papers these days (e.g., see Table 1 in [1] and [2]), can the author provide a table comparing regret bounds, constraint violation, and sample complexity of various algorithms in the related work/ introduction section? Only mentioning that the existing algorithm provides sublinear regrets (lines 40-41) does not give a full picture when the primary contribution of the paper is improving pre-constants in regret bounds.\n- Would it be better to use some other symbol instead of $T$ for the number of primal-dual updates? Since $T$ is usually used for time ($HK$) in regret bounds, it can be potentially confusing.  \n- The title is perhaps too similar to [3]. I recommend that the authors consider some modifications that highlight the contribution of this paper, particularly compared to [3].\n\n- While most of the paper is well-written and easy to follow, can the authors explain the \"profile\" mentioned in lines 167-168?\n- Missing values in Line 552 after \"With probability at least\"\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n**Reference**\n\n[1]  Tao Liu, Ruida Zhou, Dileep Kalathil, P. R. Kumar, and Chao Tian. Learning policies with zero\nor bounded constraint violation for constrained MDPs. In Proceedings of the 35th International\n Conference on Neural Information Processing Systems, NIPS ’21, 2021.\n\n[2] Gen Li, Laixi Shi, Yuxin Chen, Yuantao Gu, and Yuejie Chi. Breaking the sample complexity\n barrier to regret-optimal model-free reinforcement learning. In Proceedings of the 35th International Conference on Neural Information Processing Systems, NIPS ’21, Red Hook, NY,\n USA, 2021.\n\n[3] Sharan Vaswani, Lin Yang, and Csaba Szepesvári. Near-optimal sample complexity bounds\nfor constrained MDPs. In Advances in Neural Information Processing Systems, 2022\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

comments: SUMMARY:
The authors improve upon known results from constrained reinforcement learning. Specifically, by improving the traditional bonus-driven approach with a primal-dual intuition, the authors improve the sample complexity for learning policies in the no-violation as well as the small-violation regimes.

STRENGTHS AND WEAKNESSES:
The paper's main strength is the construction of an algorithm with improved sample complexity guarantees. In addition to the theoretical advancements, using primal-dual intuitions within the bonus-driven approach is a very nice idea. The main weakness of this paper is the presentation; the paper is very mathematically driven and terse. It could benefit from some improvements in writing for clarity. It could also be argued that the improvements in the sample complexity are not drastic, but I believe they are very meaningful nonetheless.

QUESTIONS:
Can your results be stated in another way that avoids dependency on the stater coefficient? Also, do the authors have any thoughts on the optimality of their results? Specifically, for the $\epsilon$-violating regime, would it be possible to achieve the same sample condition while achieving even smaller than $\epsilon$ violation?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 2
IMPACT: 4
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors tackle safety in RL through the framework of constrained MDPs (CMDPs). A CMDP is essentially a Markov decision process equipped with a cost function. Safety is encoded through a bound on the maximum cost the agent must incur while maximizing its return. Precisely, the authors develop a theoretical framework for tabular episodic MDPs, equipped with sample complexity for regret bounds and constraints violations. They consider two settings: strict feasibility, where the safety bound needs to be strictly satisfied, and relaxed feasibility, where the user fixes an acceptable error term violating the safety bound. Notably, the authors provide an online algorithm. This is in contrast to previous work that relied on access to a generative model (an assumption hardly feasible in practice). To achieve the guarantees, the technique is based on a primal-dual formulation. Interestingly, the provided sample complexity in the relaxed feasibility setting matches the lower bound for unconstrained MDPs, while in the strict setting, it matches the lower bound for constrained MDPs with access to a generative model.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n- The theoretical results seem rather significant: in the relaxed setting, learning in a CMDP is as easy as learning in a classic MDP. On the other hand, in the strict setting, learning in the online setting is as easy as with generative access.\n- The results presented cover two crucial axes: regret bound and constraint violation, for the two settings discussed above.\n- The authors did a good job in presenting the related work, clearly exposing the problem setup and discussing why previous methods failed to achieve near-optimal sample complexity (beginning of Section 4).\n\n# Weaknesses\n- The main concerns I have about this paper are the lack of clarity and problems of presentation. In particular, this makes the paper very difficult to follow, especially for a non-expert reader in RL sample complexity/regret analysis.\n- The paper is very dense (in particular, starting from Section 4) and extremely notation-heavy. Of course, this is a theoretical paper, and that’s the kind of work where formalities are necessary. However, finding the right balance between clarity and formality is key; in the main text, clarity is perhaps even more important as formalities, and notations make the paper very difficult to follow. Beyond the high number of symbols to digest, some of them are even reused for different purposes. For instance, $b$ is used both for the safety bound and for the UCB-style bonus. Taken out of context, it is very confusing. Another example is in Equation 8, where the authors write in a single line that the primal update is equal to subtracting the dual variable multiplied by the optimistically biased cost estimated. First, this is entirely written in the equation through subscript of the value function. This is very difficult to parse. Furthermore, this is written via several layers of subscripts, which makes it even more difficult to read. Why not detail this equality in the main text? Second, the authors do not explain why this equality holds. It should be made clearer. \n- Starting at page 5 (in the methodology section), the paper starts to become very difficult to follow due to the high number of notations, subscripts, superscripts, and the lack of motivation when introducing new concepts and algorithmic choices. Those are later on justified in the proofs, but to me, some of them should be motivated beforehand. To be clear, the design choice of the algorithms is rightfully _described_, but rarely motivated. As an example, the authors describe a double batch update scheme to learn an empirical transition matrix and briefly mention that this is to make $\\hat{V}$ independent from $\\hat{P}$. How? This is not justified. Another example is the reason why a mixture of policies is taken in Algorithm 1. This is not motivated before diving into the proofs.\n- If I am not mistaken, the algorithm assumes knowledge of the reward and cost functions (they are given as input of Algorithm 1). Unless I am mistaken, this is not clearly stated in the main text. I think this also deserves a (at least brief) discussion, even if it is considered ‘standard’ in theoretical RL — in practice, that’s not. \n- The paper ends abruptly with a Theorem and its proof. I would really have liked to have a final formal discussion about the results presented in Section 5 and their implications on the ongoing research.\n\nQUESTIONS:\n- Could you explain why ‘fixing a profile that keeps track of visitation counts for all $(s, a, h)$ tuples makes $\\hat{V}$ independent of $\\hat{P}$’ (line 168)?\n- Could you also explain how the double-batch update allows to avoid the number of profiles to be exponential in $K$?\n- Could you please motivate a bit more the introduction of the USB/Bernstein-style bonus $b^{k, \\pi}_{h, g}$ that is added to the reward function and subtracted from the cost estimate? Intuitively, how does this encourage exploration? And how does this allow us to underestimate the cumulative cost?\n- Could you comment and explain the equality of Equation 8? \n- In Algorithm 1, how do you obtain/compute exactly $\\hat{\\pi}_{t}^{k}$? We assume here to retrieve the policy maximizing Equation 8 (through a careful discretization of $\\lambda$). In practice, I guess one should use an LP formulation or policy iteration, right?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}