Title: Provably Efficient RL under Episode-Wise Safety in Constrained MDPs with Linear Function Approximation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Reinforcement Learning Theory
Abstract: We study the reinforcement learning (RL) problem in a constrained Markov decision process (CMDP), where an agent explores the environment to maximize the expected cumulative reward while satisfying a single constraint on the expected total utility value in every episode. While this problem is well understood in the tabular setting, theoretical results for function approximation remain scarce. This paper closes the gap by proposing an RL algorithm for linear CMDPs that achieves $\widetilde{\mathcal{O}}(\sqrt{K})$ regret with an episode-wise zero-violation guarantee. Furthermore, our method is computationally efficient, scaling polynomially with problem-dependent parameters while remaining independent of the state space size. Our results significantly improve upon recent linear CMDP algorithms, which either violate the constraint or incur exponential computational costs.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies linear Constrained Markov Decision Processes (CMDPs) in the function approximation setting. Concretely, the authors present two algorithms with sublinear regret which is guaranteed to never violate the constraints, one for the bandit setting and one for the general reinforcement learning setting. This is achieved by using a known safe policy to select actions whenever the confidence bounds exceed a given threshold. Otherwise the algorithm plays a policy whose occupancy measure mixes that of the safe policy and that of the estimated best policy. The authors show that the number of episodes playing the safe policy is logarithmically bounded, ensuring sublinear regret.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is mostly well written and the authors attempt to explain all steps of the analysis. According to the authors, theirs is the first known algorithm with sublinear regret and zero violation in the linear CMDP setting. I believe the algorithm and the analysis are novel, particularly the way that the known safe policy is used to achieve a mixture policy that is guaranteed to be safe. The authors also bound the number of times that the safe policy is played on its own, which is sufficient to ensure sublinear regret.\n\nThe analysis is somewhat difficult to follow at times, though I have done my best to ensure that the proofs are correct. A couple of algorithmic concepts are not well explained, e.g. how to solve a given optimistic-pessimistic problem efficiently when constraints are involved, and how to produce a policy that matches a certain occupancy measure. However, overall I am happy with the paper and assuming that all results are correct I think the contribution is significant enough for publication at NeurIPS.\n\nQUESTIONS:\nHow does Algorithm 1 solve the optimistic-pessimistic problem? This does not seem trivial due to the constraint.\n\nA policy with a given occupancy measure is guaranteed to exist, but how easy it is to compute such a policy?\n\nLIMITATIONS:\nI do not think that there are any major limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies constrained RL problem where it maximizes the reward subject to a constraint. The settings considered in this paper are episode-wise safety and linear function approximation. It designs a computationally efficient algorithm and proves that with high probability the algorithm incurs $\\tilde{O}(\\sqrt{K})$ regret with an episode-wise zero-violation guarantee.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\nWhile prior works established such episode-wise safety with sublinear regret in tabular settings, extending these guarantees to function approximation remained open. Previous work use LP based methods which is computationally inefficient with large state space. This work design and analyze an algorithm which is computationally efficient under the linear setting.\n\nWeaknesses:\n\n-Although the regret is optimal in its dependence on K, the dependence on both d and H deviates a lot from the optimal dependence. Given existing results on episode-wise constrained RL and hard-constrained linear RL, the main advance seems to be computational rather than statistical (since it sacrifices on the sample complexity). Although the authors mentioned that adapting Bernstein-type bonus can improve it, but this improvement (H) is not enough to cover the loss.\n\n-There are many existing work on constrained/safe RL with good convergence rates. Although the episode-wise safety setting considers a stricter case, it relies on an unrealistic assumption on having access to a known baseline policy that is strictly safe. Note that this assumption is not common in other constrained RL settings. This limits the practical impact of the guarantees.\n\n-This paper studies linear function approximation setting, which is common for large state-action space RL problem. However, the empirical experiments are done in a small size CMDP.\n\nQUESTIONS:\nCould you be more specific about the claim on “while safe exploration is well-established in tabular CMDPs, extending it to large-scale CMDPs remains a major challenge”? Given existing work with optimistic–pessimistic exploration techniques and safe algorithms with linear function approximation, could you articulate the specific challenge your paper resolves? This is a solid work but the novelty is not yet clear to me.\n\nSome minor suggestions:\n\nProviding a table of content would improve readability.\n\nA small typo in Assumption 2: \\mu_h(s’) should be replaced with (\\mu_h)\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates policy optimization methods where the policy must satisfy the constraint for all iterations in a linear CMDP.\nGiven access to a policy that strictly satisfies the constraints, the author proposed a method that achieves $O(\\sqrt{d^5 H^8 K}$ regret with high probability. To achieve this, the author add standard exploration bonuses to the rewards and utilities but provides a novel condition when to switch to $\\pi^{\\text{sf}}$. Additionally, the authors conduct experiments in the tabular setting to support their theoretical results.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. This paper provide solid theoretical results for a challenging problem instance and even recover similar regret bounds to prior work in the absence of constraints.\n2. The paper is well organized, and it was to first consider the problem in a simpler bandit setting before moving to CMDPs.\n3. The empirical experiments validate the theoretical results on (a bit simplistic) environments.\n\n**Weakness**\n1. While tabular CMDPs are linear CMDP,  it would be great to see experiments for linear CMDP. Otherwise, it would be interesting to see how the OPLB-SP compares to DOPE [1] in the tabular setting. I would strength my score for either addition.\n2. It seems like the parameters of Algorithm 2 appear to be specified only up to big-O notation. It would be helpful to clarify what aspects of the analysis prevents the exact values to be known.\n\n[1] Bura, A., HasanzadeZonuzy, A., Kalathil, D., Shakkottai, S., & Chamberland, J. F. (2022). DOPE: Doubly optimistic and pessimistic exploration for safe reinforcement learning. Advances in neural information processing systems, 35, 1047-1059.\n\nQUESTIONS:\n1. Is it a strict requirement that the initial state $s_1$ is fixed, or can the results be extended when $s_1$ may be chosen adversarially? \n2. In Figure 1, the average number of $\\pi^{\\text{safe}$ deployments appears consistent across different random seeds initially, but then begins to diverge around the midpoint of training as shown by the widening shaded confidence intervals (I'm guessing that this is what the shaded regions depict). What could explain this pattern of initial convergence followed by increasing variance between seeds?\"\n3. In Line 4 of Algorithm 1, how it be guaranteed to a $\\pi^{k}$ to always exist? It wasn't clear to me why Lemma 24 implies this fact.\n\nLIMITATIONS:\nYes, the authors sufficiently address the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper investigates the constrained MDP problems with zero-violation guarantee. The main contribution are the algorithms for linear bandit and linear CMDPs based on the linear function approximation and the optimism-pessimism exploration method. The novelty of the algorithm lies in the a new deployment rule and the composite softmax policy. The algorithms are also computationally efficient, achieving a polynomial computational cost in parameters and independent of the state space. The authors further prove that the algorithm is the first result achieving zero-violations and sub-linear $\tilde O(\sqrt{K})$ regret.

STRENGTHS AND WEAKNESSES:
**Strengths**:
1. The algorithm is novel and is the first result that achieving $\tilde O(\sqrt{K})$ regret and zero-violation.
2. The algorithm achieves the polynomial computational cost on the parameters and is independent of state-space.
3. Numerical experiments further validate the performance of the algorithms.

**Weaknesses**:
1. The algorithm is only suitable for the one-constraint setting; there are often multiple constrains in real problems
2. The computational time is still dependent on action space, and the algorithm will still be computationally inefficient when meets the large or infinite action space; Bisection search may also time-consuming when the space is large.
3. The regret has higher dependence on parameters $d,H$ compared to other similar algorithms.
4. The paper could be clearer; it's a bit messy

QUESTIONS:
Can authors add the comparison with Roknilamouki's algorithm? It seems that their algorithm has similar theoretical results and also zero-violation.

LIMITATIONS:
See the Weakness in **Strengths And Weaknesses** part.
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-05