Title: Taming Adversarial Constraints in CMDPs
Venue: NeurIPS 2025
Accepted: yes
Keywords: CMDP

Online Learning

Constraints
Abstract: In constrained MDPs (CMDPs) with adversarial rewards and constraints, a known impossibility result prevents any algorithm from attaining sublinear regret and constraint violation, when competing against a best-in-hindsight policy that satisfies the constraints on average. In this paper, we show how to ease such a negative result, by considering settings that generalize both stochastic CMDPs and adversarial ones. We provide algorithms whose performances smoothly degrade as the level of environment adverseness increases. In this paper, we show that this negative result can be eased in CMDPs with non-stationary rewards and constraints, by providing algorithms whose performances smoothly degrade as non-stationarity increases. Specifically, they attain $\widetilde{\mathcal{O}} (\sqrt{T} + C)$ regret and positive constraint violation under bandit feedback, where $C$ measures the adverseness of rewards and constraints. This is $C = \Theta(T)$ in the worst case, coherently with the impossibility result for adversarial CMDPs. First, we design an algorithm with the desired guarantees when $C$ is known. Then, in the case $C$ is unknown, we obtain the same results by embedding multiple instances of such an algorithm in a general meta-procedure, which suitably selects them so as to balance the trade-off between regret and constraint violation.


Peer reviews:
comments: SUMMARY:
The paper studies RL for constrained MDP where both the rewards and the constraints are adversarial. The paper bound the regret and (positive) constraint violation in terms of the corruption budget $C$. There is a related corruption-robust work under unconstrained MDP, but the setting for constrained MDP is largely unexplored. The paper first presents an optimistic algorithm when the corruption budget is known in advance. Then, they introduce a meta algorithm, reminiscent of the corralling bandit algorithm, that runs a meta bandit algorithm over multiple optimistic algorithms tuned for different corruption budgets.

STRENGTHS AND WEAKNESSES:
Strengths:

The problem setup and result are novel. The presentation of their algorithm is clear and mostly easy to follow.

Weaknesses:

The algorithm seems to be similar to the one introduced by Jin et al. (2024). Highlighting the difference in the main paper seems necessary.

QUESTIONS:
Q1) The corralling bandit reduction idea in corruption setting seems to have been also used in Jin et al. 2024 (Algorithm 3 in their paper uses similar FTRL). Although Jin et al study a different setting (adversarial loss + corrupted transitions), I think it would be good to discuss their algorithm and highlight the novelty of your algorithm in comparison to their algorithm in detail. What is the main difference from their algorithm?

Q2) The paper studies corrupted reward + corrupted cost setting. Have you considered extending to corrupted reward + corrupted cost + corrupted transition?

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-06-24

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe author studies a constrained Markov decision process (CMDP) setting in which both the rewards and constraints are sampled from probability distributions that may change adversarially across episodes. This setting differs from the standard CMDP framework, which usually assumes that these distributions are fixed or stationary. To address this more general and challenging setting, the author proposes an approach based on occupancy measures and provides theoretical performance guarantees. Specifically, the paper considers two distinct cases: one where the nature of the adversarial corruption is known, and one where it is unknown. For each case, a dedicated algorithm is developed to ensure valid and efficient decision-making under uncertainty.\n\nSTRENGTHS AND WEAKNESSES:\n\\textbf{Strengths.} The paper is well-structured, and the theoretical analysis is solid and convincing. This work addresses a challenging setting in which both the rewards and constraints are sampled from probability distributions that may change adversarially across episodes. Such a setting is difficult to handle and is not supported by many existing approaches in the literature.\n\n\\textbf{Weaknesses.} Although the theoretical guarantees are strong, the paper would benefit from empirical results. Including experiments would help illustrate the practical performance of the proposed algorithms and provide a more direct understanding of their behavior in realistic settings.\n\nQUESTIONS:\n1. While it is common to use occupancy measures in discrete settings, they can also be applied in continuous state or action spaces. Can the proposed framework be extended to handle continuous CMDPs? If so, what are the main challenges or limitations in doing so?\n\n2. For CMDP problems, a practical way to evaluate the performance of an algorithm is to examine the level of constraint violation in the empirical results. In particular, it is useful to study the trade-off between the objective value and the degree of constraint violation in out-of-sample testing. Can the author provide experimental results to demonstrate this trade-off and compare the performance of the proposed method against relevant baselines?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies a CMDP where both rewards and constraints can face adversarial corruptions, whose cumulative magnitudes are denoted by $C_r$ and $C_G$ resp. Under unknown-transition and bandit-feedback model, an occupancy-measure--based algorithm with regret & positive constr violation of order $\\tilde{\\mathcal O}(\\sqrt T+\\max(C_r,C_G))$ is proposed. When no corruptions are present, i.e., a classical stochastic CMDP instance, the bound translates to $\\tilde{\\mathcal O}(\\sqrt T)$; in the worst case when rewards and constraints are completely adversarial, the bound agrees with the $\\Omega(T)$ lower bound.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. The paper is overall well-written, with a clear flow on related works, main technical challenges, and the two steps towards $(\\sqrt T+C)$ regret.\n2. The $(\\sqrt T+C)$-style results is tight (up to logs), which distinguishes itself from the typical $\\sqrt{C}$-style dependency in classical bandits/MDPs and highlights the difficulty of adversarialness in CMDPs, as pointed out by the $\\Omega(T)$ lower bound. \n3. The technical contribution regarding the extension of CORRAL is non-trivial -- while the online learning loss / updates in Eqs. (5) and (6) do not look pretty astonishing on their own, according to the authors there is no suitable CORRAL-variants that work for primal-dual--tyle problems. But I'm not fully certain that I understand this part correctly; please refer to Weakness 2.\n\nWeakness:\n1. For the known-$C$ case, the algorithm is more-or-less standard adaptation of a stochastic algorithm to its corrupted counterpart. Nevertheless, given the extension to unknown-$C$ case, I understand that Sec 3 is more like a building block rather than main contribution.\n2. The authors mentioned that Lag-FTRL can be of independent interest; however, the current algorithmic form and description lack sufficient explanations to allow a seamless extension. It would be much better if the authors could write something as \"suppose that we'd like to solve a primal-dual problem like xxx with primal objective aaa and dual objective bbb, then instead of standard CORRAL we can use a Lagrangian-based variant of CORRAL that does ... s.t. aaa and bbb are optimized\".\n3. The transitions must be stationary (but I understand that it would be substancially harder for occupancy-measure--based algorithms). Nevertheless, it would be very interesting to incorporate the non-stationary(--transition) MDP literature.\n\nMinor: Lines 68--73 don't read to different from CORRAL / its variants; it would be better to move Lines 100--104 earlier.\n\nBut overall, given the clearly-written draft, the strong regret guarantee, and the non-trivial technical contribution, I would love to recommend an acceptance.\n\nQUESTIONS:\nSee Weaknesses 1 -- 3.\n\nAdditional question: Does it matter to change $OPT_{\\bar r,\\bar G,\\alpha}$ to $OPT_{r^\\ast,G^\\ast,\\alpha}$ where $r^\\ast,G^\\ast$ are the argmin in defns of $C_r$ and $C_G$? I think no?\n\nAlso in Lag-FTRL -- why we don't need to dynamically maintain a Lagrangian during the algorithm, like a standard primal-dual--based online learning algorithm? That is, why an upper bound on the Lagrangian $\\Lambda$ would suffice?\n\nLIMITATIONS:\nNot really. But aside from the Weakness 3 which is a minor one, I do not see many obvious limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nConstrained MDPs have traditionally been studied under two separate settings: with either stochastic or adversarial rewards and constraints. This paper designs algorithms with bandit feedback that can handle both, allowing rewards and constraints to change non-stationarily across episodes. The degree of adversariality is characterized by what the authors call the adversarial corruption C (essentially, the L1 deviation of the expected rewards and constraints). The first part of the paper introduces an optimism-based algorithm achieving $O(\\sqrt T+C)$ regret when C is known. \nThe second part develops an FTRL-based meta-algorithm that achieves the same regret guarantees even when C is unknown, by running multiple instances of the optimistic algorithm for different candidate values of C.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is well written overall, and I appreciate how the authors point out relevant connections to existing work. Conceptually, this paper shows that one can handle positive constraints (that is, constraints cannot cancel each other at different times), and shows that one interpolate between stochastic and adversarial setups with a regret bound that is analogous to previous works on corruption robust online learning, which is of interest to the community.\n\nThe proof techniques build significantly over previous works, especially Jin et al. 2024. The proof of the first result when the corruption level C is known seems fairly standard (handling the corruption level is mainly done by constructing appropriate confidence intervals). The main technical contribution seems to be for the second result when C is unknown. The high-level procedure is quite simple: instantiate (log T) several algorithms for exponential scales of C and run some FTRL procedure on it, however the proof requires bounding the size of the dual variable to enforce constraints and using a stabilizing technique from Jin et al. 2024. Overall the novelty seems sufficient.\n\nOne area that could be strengthened is the motivation. The authors give concrete examples in introduction for constrained MDPs, which are relatively well established, but could spend some time motivating adversarial rewards and constraints for the examples for constrained MDPs. \n\nThe benchmark should also be further motivated, since it is not standard and not fully consistent with the algorithms. Specifically, the violation total cost $V_T$ counts each violation positively: $V_T = \\max\\_{i\\in[m]} \\sum_{t\\in[T]} [\\mathbb E [G_t]^\\top q_t-\\alpha]_{i}^{+}$,\n\n but the benchmark for the regret is a policy that only satisfies these constraints in a non-positive way (that is, cancellations are allowed): $R_T=\\sum_{t\\in[T]} \\mathbb E[r_t]^\\top (q^\\star - q_t)$ where $q^\\star$ is only constrained to satisfy $\\bar G^\\top q^\\star \\leq \\alpha$, where $\\bar G = \\frac{1}{T}\\sum_{t\\in[T]} \\mathbb E[G_t]$. This is of course a stronger benchmark that if we were to ask that $\\mathbb E[G_t]^\\top q^\\star \\leq \\alpha$ for all $t\\in[T]$ but it would seem natural to impose the latter (or similarly to agnostic-type bounds, count the violation of the benchmark in a positive way) and then perhaps achieve stronger bounds (if possible). In that case, is it possible to achieve better bounds? Does the linear regret T bound still hold in a fully adversarial case for this milder benchmark?\n\nLast, the paper would be strenghened by discussing further potential lower bounds. In particular is $O(\\sqrt T + C)$ regret and violation necessary? Are there any known lower bounds (e.g. reductions from corruption-robust online learning? What about the dependencies in terms of $|X|,|A|,m$, are these tight?\n\nQUESTIONS:\nSome questions above.\n\n- Is there any tradeoff between regret and violation? In particular, if we do not have lower bounds $O(\\sqrt T + C) $, can we at least show that Algorithm 2 or 3 are Pareto-optimal?\n\n- In the introduction, I wasn’t sure how the second application about recommender systems (ensuring users are not exposed to offensive content) fits into this framework -- how are the constraints non-stationary?\n\nMinor comments:\n\np3 Section 2. Define formally $L$ as the length of each episode.\n\np4 l162. $X_k$ was only defined in the footnote, it would be preferable to have it defined formally in the main body.\n\np8 l326. At this point, $w_{t,j}$ was not yet introduced in the text.\n\nLIMITATIONS:\nLimitations discussed appropriately.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-08"}