Title: Provable Distributional Value Iteration under Partial Observability
Primary area: learning theory
Keywords: Reinforcement Learning, Partial Observability, Distributional Reinforcement Learning, POMDP, Planning

Abstract: In many real-world planning tasks, agents must tackle uncertainty about the environment’s state and variability in the outcomes of any chosen policy. We address both forms of uncertainty as a first step toward safer algorithms in partially observable settings. Specifically, we extend Distributional Reinforcement Learning (DistRL)—which models the entire return distribution for fully observable domains—to Partially Observable Markov Decision Processes (POMDPs), allowing an agent to learn the distribution of returns for each conditional plan. Concretely, we introduce new distributional Bellman operators for partial observability and prove their convergence under the supremum $p$-Wasserstein metric. We also propose a finite representation of these return distributions via $\psi$-vectors, generalizing the classical $\alpha$-vectors in POMDP solvers. Building on this, we develop Distributional Point-Based Value Iteration (DPBVI), which integrates $\psi$-vectors into a standard point-based backup procedure—bridging DistRL and POMDP planning. Our experiments demonstrate that DPBVI matches classical Point-Based Value Iteration (PBVI) in the risk-neutral case, validating the distributional extension.

TL;DR: First theoretical framework for distributional RL in POMDPs: new distributional operators, finite ψ-vector representation, and DPBVI, a point-based algorithm for robust planning under partial observability.