NeurIPS 2025

Online Optimization for Offline Safe
Reinforcement Learning
Yassine Chemingui
Washington State University
yassine.chemingui@wsu.edu
Aryan Deshwal
University of Minnesota
adeshwal@umn.edu
Alan Fern
Oregon State University
alan.fern@oregonstate.edu
Thanh Nguyen-Tang
New Jersey Institute of Technology
thanh.nguyen@njit.edu
Janardhan Rao Doppa
Washington State University
jana.doppa@wsu.edu
Abstract
We study the problem of Offline Safe Reinforcement Learning (OSRL), where the
goal is to learn a reward-maximizing policy from fixed data under a cumulative
cost constraint. We propose a novel OSRL approach that frames the problem as a
minimax objective and solves it by combining offline RL with online optimization
algorithms. We prove the approximate optimality of this approach when integrated
with an approximate offline RL oracle and no-regret online optimization. We
also present a practical approximation that can be combined with any offline RL
algorithm, eliminating the need for offline policy evaluation. Empirical results
on the DSRL benchmark demonstrate that our method reliably enforces safety
constraints under stringent cost budgets, while achieving high rewards. The code is
available at https://github.com/yassineCh/O3SRL.
1
Introduction
Offline reinforcement learning (RL) [Levine et al., 2020] is a powerful paradigm to learn decision-
making policies from logged datasets without the need for additional interaction with the environment.
Offline RL has shown good success in domains including autonomous driving, robotics, control
systems, and black-box optimization [Fang et al., 2022, Lin et al., 2024, Li et al., 2024, Zhan et al.,
2022, Chemingui et al., 2024] where executing exploratory actions is not practical. However, in
safety-critical domains such as healthcare and smart grid, the decision-making agent needs to also
satisfy some cost/safety constraints. Such problems are studied under the sub-area referred to as
offline safe RL (OSRL) where the goal is to learn r...