ICML 2025

Provable Policy Gradient for Robust Average-Reward MDPs
Beyond Rectangularity
Qiuhao Wang * 1 Yuqi Zha * 2 Chin Pang Ho 2 Marek Petrik 3
Abstract
Robust Markov Decision Processes (MDPs) of-
fer a promising framework for computing reliable
policies under model uncertainty. While policy
gradient methods have gained increasing popular-
ity in robust discounted MDPs, their application
to the average-reward criterion remains largely
unexplored. This paper proposes a Robust Pro-
jected Policy Gradient (RP2G), the first generic
policy gradient method for robust average-reward
MDPs (RAMDPs) that is applicable beyond the
typical rectangularity assumption on transition
ambiguity. In contrast to existing robust policy
gradient algorithms, RP2G incorporates an adap-
tive decreasing tolerance mechanism for efficient
policy updates at each iteration. We also present a
comprehensive convergence analysis of RP2G for
solving ergodic tabular RAMDPs. Furthermore,
we establish the first study of the inner worst-
case transition evaluation problem in RAMDPs,
proposing two gradient-based algorithms tailored
for rectangular and general ambiguity sets, each
with provable convergence guarantees. Numerical
experiments confirm the global convergence of
our new algorithm and demonstrate its superior
performance.
1. Introduction
Markov Decision Processes (MDPs) (Puterman, 2014) pro-
vide a powerful framework for sequential decision-making,
with applications such as game solving (Mnih, 2013), health-
care (Shechter et al., 2008), and finance (Bäuerle & Rieder,
*Equal contribution 1Fintech Innovation Center, Research In-
stitute for Digital Economy and Interdisciplinary Sciences, South-
western University of Finance and Economics 2Department of
Data Science, City University of Hong Kong 3Department of
Computer Science, University of New Hampshire. Correspon-
dence to: Chin Pang Ho <clint.ho@cityu.edu.hk>, Marek Petrik
<mpetrik@cs.unh.edu>.
Proceedings of the 42 nd International Conference on Mach...