ICML 2025

Robust Offline Reinforcement Learning with Linearly Structured
f-Divergence Regularization
Cheng Tang * 1 Zhishuai Liu * 2 Pan Xu 2
Abstract
The Robust Regularized Markov Decision Pro-
cess (RRMDP) is proposed to learn policies ro-
bust to dynamics shifts by adding regularization
to the transition dynamics in the value function.
Existing methods mostly use unstructured reg-
ularization, potentially leading to conservative
policies under unrealistic transitions. To address
this limitation, we propose a novel framework,
the d-rectangular linear RRMDP (d-RRMDP),
which introduces latent structures into both tran-
sition kernels and regularization. We focus on
offline reinforcement learning, where an agent
learns policies from a precollected dataset in the
nominal environment. We develop the Robust
Regularized Pessimistic Value Iteration (R2PVI)
algorithm that employs linear function approxi-
mation for robust policy learning in d-RRMDPs
with f-divergence based regularization terms on
transition kernels. We provide instance-dependent
upper bounds on the suboptimality gap of R2PVI
policies, demonstrating that these bounds are in-
fluenced by how well the dataset covers state-
action spaces visited by the optimal robust policy
under robustly admissible transitions. We estab-
lish information-theoretic lower bounds to verify
that our algorithm is near-optimal. Finally, numer-
ical experiments validate that R2PVI learns robust
policies and exhibits superior computational effi-
ciency compared to baseline methods.
1. Introduction
Offline reinforcement learning (RL) (Levine et al., 2020)
facilitates policy learning from fixed datasets, eliminating
the need for direct interaction with the environment. When
*Equal contribution 1 University of Illinois Urbana-Champaign,
work was done when Cheng was in Tsinghua University 2Duke
University. Correspondence to: Pan Xu <pan.xu@duke.edu>.
Proceedings of the 42 st International Conference on Machine
Learning, Vancouver, Canada. PMLR 267...