Title: Off-Policy Safe Reinforcement Learning with Cost-Constrained Optimistic Exploration
Primary area: reinforcement learning
Keywords: constrained reinforcement learning, safe reinforcement learning, safe exploration, epistemic uncertainty quantification

Abstract: When formulating safety as limits of cumulative cost, safe reinforcement learning (RL) learns policies that maximize rewards subject to these constraints during both data collection and deployment. While off-policy methods offer high sample efficiency, their application to safe RL faces substantial challenges from constraint violations caused by estimation bias in value functions and cost-agnostic exploration. To address these challenges, we propose Constrained Optimistic eXploration Q-learning (COX-Q), an off-policy primal-dual safe RL method that integrates cost-bounded exploration and conservative distributional RL. 
First, we introduce a novel cost-constrained optimistic exploration strategy that resolves gradient conflicts between reward and cost in the action space, and adaptively adjusts the trust region to control constraint violation in exploration. Second, we adopt truncated quantile critics to mitigate estimation bias. The quantile critics also quantify distributional, risk-sensitive epistemic uncertainty for guiding exploration. Experiments across velocity-constrained robot locomotion, safe navigation, and complex autonomous driving tasks demonstrate that COX-Q achieves high sample efficiency, competitive safety performance during evaluation, and controlled data collection cost in exploration. The results highlight the proposed method as a promising solution for safety-critical applications of RL agents.

TL;DR: An off-policy primal-dual safe reinforcemen learning approach with cost-bounded optimistic exploration.