Title: Towards Learning POMDPs Without Full Observability
Primary area: applications to robotics, autonomy, planning
Keywords: Learning for planning under uncertainty, POMDPs

Abstract: We are interested in enabling autonomous agents to learn and reason about systems with hidden states, such as furniture with hidden locking mechanisms.
We cast this problem as learning the parameters of a discrete Partially Observable Markov Decision Process (POMDP).
The agent begins with knowledge of the POMDP's actions and observation spaces, but not its state space, transitions, or observation models. 
These properties must be constructed from action-observation sequences. 
Spectral approaches to learning models of partially observable domains, such as learning Predictive State Representations (PSRs), are known to be able to directly estimate the number of hidden states. 
These methods cannot, however, yield direct estimates of transition and observation likelihoods, which are important for many downstream inference and learning tasks. 
Other approaches leverage tensor decompositions to estimate transition and observation likelihoods but often assume full state observability and full-rank transition matrices for all actions.
To relax these assumptions, we study how PSRs learn transition and observation matrices up to a similarity transform, which may be estimated via tensor methods.
Our method learns observation matrices and transition matrices up to a partition of states, where the states in a single partition have the same observation distributions corresponding to actions whose transition matrices are full-rank.
Our experiments suggest that these partition-level transition models learned by our method, with a sufficient amount of data, meets the performance of PSRs as models to be used by standard sampling-based POMDP solvers but yields proper probabilistic models leveragable by downstream inference operations.

TL;DR: POMDP, learning for planning under uncertainty