Title: Generalization in Monitored Markov Decision Processes (Mon-MDPs)
Primary area: reinforcement learning
Keywords: Reinforcement learning, Monitored-MDPs, Partially observable rewards, Generalization, Function approximation, Caution, Partial Observability, RL Safety, DQN, CVaR, Reward Model

Abstract: Reinforcement learning (RL) typically models the interaction between the agent and environment as a Markov decision process (MDP), assuming fully observable rewards. In many real-world settings, this assumption fails, motivating the monitored Markov decision process (Mon-MDP), where rewards may be unobserved. Prior work on Mon-MDPs has been limited to simple, tabular cases, restricting their applicability to real-world problems. This work explores Mon-MDPs using function approximation and investigates the challenges. We show that combining function approximation with a learned reward model enables agents to generalize from monitored states with observable rewards to unmonitored states with unobservable rewards. Therefore, we demonstrate that such generalization with a reward model achieves near-optimal policies in environments formally defined as unsolvable. However, we also uncover a critical limitation: agents may incorrectly extrapolate rewards due to *overgeneralization*, which can lead to undesirable behaviors. To mitigate overgeneralization, we propose a cautious policy optimization method leveraging reward uncertainty. This work serves as a step towards bridging the gap between Mon-MDP theory and real-world applications.

TL;DR: Extend Monitored-MDPs to non-tabular environments using function approximation and investigates the associated challenges.