NeurIPS 2025

On Evaluating Policies for Robust POMDPs
Merlijn Krale∗
Radboud University
Nijmegen, The Netherlands
merlijn.krale@ru.nl
Eline M. Bovy∗
Radboud University
Nijmegen, The Netherlands
eline.bovy@ru.nl
Maris F. L. Galesloot∗
Radboud University
Nijmegen, The Netherlands
maris.galesloot@ru.nl
Thiago D. Simão
Eindhoven University of Technology
Eindhoven, The Netherlands
t.simao@tue.nl
Nils Jansen
Ruhr-University Bochum & Radboud University
Bochum, Germany & Nijmegen, The Netherlands
n.jansen@rub.de
Abstract
Robust partially observable Markov decision processes (RPOMDPs) model sequen-
tial decision-making problems under partial observability, where an agent must
be robust against a range of dynamics. RPOMDPs can be viewed as a two-player
game between an agent, who selects actions, and nature, who adversarially selects
the dynamics. Evaluating an agent policy requires finding an adversarial nature
policy, which is computationally challenging. In this paper, we advance the eval-
uation of agent policies for RPOMDPs in three ways. First, we discuss suitable
benchmarks. We observe that for some RPOMDPs, an optimal agent policy can be
found by considering only subsets of nature policies, making them easier to solve.
We formalize this concept of solvability and construct three benchmarks that are
only solvable for expressive sets of nature policies. Second, we describe a new
method to evaluate agent policies for RPOMDPs by solving an equivalent MDP.
Third, we lift two well-known upper bounds from POMDPs to RPOMDPs, which
can be used to efficiently approximate the optimality gap of a policy and serve as
baselines. Our experimental evaluation shows that (1) our proposed benchmarks
cannot be solved by assuming naive nature policies, (2) our method of evaluating
policies is accurate, and (3) the upper bounds provide solid baselines for evaluation.
1
Introduction
Partially observable Markov decision processes [POMDPs; 25] are ubiquitous for representing
sequential decision-making prob...