ICML 2025

Model-Based Exploration in Monitored Markov Decision Processes
Alireza Kazemipour 1 2 Matthew E. Taylor 1 2 3 Michael Bowling 1 2 3
Abstract
A tenet of reinforcement learning is that the agent
always observes rewards. However, this is not
true in many realistic settings, e.g., a human ob-
server may not always be available to provide
rewards, sensors may be limited or malfunction-
ing, or rewards may be inaccessible during de-
ployment. Monitored Markov decision processes
(Mon-MDPs) have recently been proposed to
model such settings. However, existing Mon-
MDP algorithms have several limitations: they
do not fully exploit the problem structure, can-
not leverage a known monitor, lack worst-case
guarantees for “unsolvable” Mon-MDPs without
specific initialization, and offer only asymptotic
convergence proofs. This paper makes three con-
tributions. First, we introduce a model-based algo-
rithm for Mon-MDPs that addresses these short-
comings. The algorithm employs two instances
of model-based interval estimation: one to ensure
that observable rewards are reliably captured, and
another to learn a minimax-optimal policy. Sec-
ond, we empirically demonstrate the advantages.
We show faster convergence than prior algorithms
in more than four dozen benchmarks, and even
more dramatic improvements when the monitor-
ing process is known. Third, we present the first
finite sample-bound on performance. We show
convergence to a minimax-optimal policy even
when some rewards are never observable.
1. Introduction
Reinforcement learning (RL) is based on trial-and-error: in-
stead of being directly shown what to do, an agent receives
consistent numerical feedback in the form of rewards for
its decisions. However, this assumption is not always re-
1Department of Computing Science, University of Alberta
2Alberta Machine Intelligence Institute (Amii) 3Canada CIFAR
AI Chair, Amii.
Correspondence to:
Alireza Kazemipour
<kazemipo@ualberta.ca>.
Proceedings of the 42 nd International Confere...