Reinforcement Learning with State Observation Costs in Action-Contingent Noiselessly Observable Markov Decision Processes 

1 Introduction . Reinforcement learning ( RL ) , which involves learning to make sequences of good decisions in stochastic environments , has had many impressive successes over the past decade [ 60 , 68 ] . While many of the more visible successes have occurred in simulated environments , such as Atari video games [ 44 , 58 ] , reinforcement learning has increasingly been used to solve real-world problems [ 10 , 46 , 36 , 41 , 75 , 47 , 37 , 51 , 42 , 34 ] . One key difference between reinforcement learning in simulated vs. real-world environments is that , in most simulated environments , the agent can fully observe the underlying state at each time step with no additional observation costs . In contrast , real-world environments frequently entail state observations that are noisy and/or costly , but optional . Environ- * Equal contribution 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . ments in which state information at each time step can be noisy or missing are more appropriately modeled as partially observable Markov decision processes ( POMDPs ) rather than MDPs [ 29 ] . In this paper , we provide theory and algorithms for a special class of POMDPs in which state information is complete/noiseless when observed , but may be missing at any given time step if the agent chooses not to observe the state . We call this class of POMDPs Action-Contingent Noiselessly Observable MDPs ( ACNO-MDPs ) , which can be useful for capturing a number of important realworld settings , such as : ACNO-MDPs in healthcare . Clinicians in Intensive Care Units ( ICUs ) frequently have to make sequences of treatment decisions under uncertainty for patients at risk . While accurate laboratory tests can inform such decisions , administration of these tests carry a significant cost for the patient and the health system [ 25 ] . These costs , together with the fact that frequent testing may be redundant and wasteful [ 12 , 78 ] , appropriately lead clinicians to refrain from constantly observing the patient state ( i.e. , ordering laboratory tests ) [ 1 , 17 ] . Similarly , other settings – such as glucose monitoring to assist in insulin dosing recommendations [ 18 ] , or white blood cell count monitoring to assist in anti-HIV drug dosing [ 15 ] – have modern tools for accurate observations of biomarkers and could be appropriately modeled by our ACNO-MDP framework . ACNO-MDPs for user-adaptive experiences . Applications on mobile phones and other personal devices can collect information on the user ’ s status , such as location , motion , inter-user contact , and background noise , in order to adaptively suggest features that enhance the user ’ s experience [ 69 , 70 ] . While these sensors can in theory always be kept active , battery consumption due to constant sensing would make such applications less desirable to the user . Modeling this problem as an ACNO-MDP could enable a policy that balances battery usage with personalizing the user ’ s experience . 1 Our contributions . Our main contributions are : • Proposing a Probably Approximately Correct “ Observe-then-Plan ” algorithm for tabular ACNO-MDPs that fully observes while exploring , then employs POMDP planning using learned models . The resulting policy can select when to observe to achieve high expected reward in environments with state observations that are costly but optional . • Providing a finite sample guarantee for the above algorithm in environments with discrete states and actions , such that the number of episodes exhibiting worse than near-optimal behavior is no larger than O ( S 5A2H5Rmax 3◆4 ✏3 ) with ◆ = O ( ln ( SAH/ ✏ ) ) with high probability . This improves over prior work targeted towards more general POMDP problems ( Table 4 ) . • Developing an algorithm that extends these ideas to ACNO-MDPs with continuous state and action spaces . We introduce a novel belief state encoder that can be coupled with many modern deep RL algorithms to solve problems with observation costs , which we demonstrate with several simulation experiments . 2 Related work . Learning in POMDPs without a known model of the environment ( i.e. , with neither a model of state-to-state transition dynamics nor a model of observations conditioned on states ) is significantly more difficult than learning in MDPs [ 49 ] . Thus while there is a rich literature on planning in POMDPs with a known environment model [ 3 , 7 , 13 , 23 , 35 , 39 , 40 , 50 , 53 , 54 , 57 , 59 , 62 , 67 , 77 ] , including in Mixed Observability MDPs ( MOMDPs ) [ 61 ] , active perception frameworks [ 19 , 63 ] , and Non-Observable MDPs ( NOMDPs ) [ 38 ] , fewer approaches exist for RL in POMDPs ( POMDPRL ) . One key difficulty in general POMDP-RL is that , without a known observation model , the agent can never know with certainty which states were actually visited . Approximating a transition model for use in subsequent planning thus becomes infeasible without additional assumptions on the observation model . Many approaches in POMDP-RL , such as the Bayes-Adaptive POMDP ( BAPOMDP ) [ 52 , 56 ] and its variants [ 31 , 32 ] , assume access to an informative prior distribution over the observation model and maintain posterior distributions over the environment model parameters . These algorithms plan using belief distributions over states rather than the states themselves . More recent 1While some of the POMDP-RL methods proposed in this paper may not be energy efficient , recent approaches for compiling and compressing policies into approximately equivalent finite-state controllers could enable energy efficient user-adaptive applications that both sense and change behaviors adaptively [ 20 ] . approaches like Deep Recurrent Q-Networks ( DRQN ) [ 22 , 79 ] and Deep Variational Reinforcement Learning ( DVRL ) [ 43 ] use neural network-based sequence models to learn a condensed representation of the agent ’ s action and observation history and apply modern RL algorithms ( e.g. , DQN [ 44 ] , A2C [ 72 ] ) on this learned representation . These approaches show promising empirical performance but lack theoretical guarantees and do not leverage the specific structure of ACNO-MDPs . Theoretical guarantees on a POMDP-RL algorithm ’ s performance can be beneficial when gathering samples under a suboptimal policy is costly . Though recent works have been able to provide theoretical guarantees in surprisingly broad classes of POMDPs , these works either provide regret guarantees ( not sample complexity guarantees ) [ 24 , 66 ] , or make assumptions about the structure of the observation matrix which do not hold in ACNO-MDPs ( e.g. , that the observation matrix is full rank [ 21 , 2 ] or its singular value is bounded away from 0 [ 27 ] ) . To the best of our knowledge , our work therefore provides sample complexity guarantees for a subclass of POMDPs in which no such prior guarantees existed . Closely related to our work , Klima et al . [ 33 ] consider RL in environments where the state is fully observed , but only intermittently . However , whereas Klima et al . [ 33 ] consider environments in which the agent can not control state observation frequency , we address RL in environments where the agent can decide when to observe , albeit at a cost . Similarly , Yoon et al . [ 74 ] and Chang et al . [ 8 ] use RL methods to learn policies for active sensing/dynamic measurement scheduling . In these works , however , the reward is directly tied to prediction accuracy and agents have no influence on the underlying state beyond observation whereas we focus on agents that can influence the state . Since acceptance , it has come to our attention that earlier work by Zubek and Dietterich [ 80 ] , Zubek et al . [ 81 ] , and Bellinger et al . [ 4 ] proposed RL approaches for learning optimal policies that both modify and observe the state at each time step , as in our problem setup . However , these prior works only provide algorithms for tabular environments and do not provide any theoretical analysis bounding the sample complexity . Related contemporaneous works by Yin et al . [ 73 ] and Bellinger et al . [ 5 ] tackle the problem of RL with observation costs in continuous-state POMDPs and MDPs , respectively : Yin et al . [ 73 ] considers a setting where individual elements of the state vector can be observed at each point in time and Bellinger et al . [ 5 ] performs a more in-depth qualitative analysis of agents ’ measurement patterns throughout training . 3 Problem setting . Action-contingent noiselessly observable Markov decision processes ( ACNO-MDPs ) . A discounted , finite-horizon ACNO-MDP is an environment defined by a tuple M = hS , A , O , p , , r , b , Hi , where S and A are state and action spaces with cardinalities S and A , respectively . The action space A consists of tuples { control actions } ⇥ { observe , not observe } . As in POMDPs , the observation space , O , has cardinality O , and is related to S by the observation/emission function p ( o|s0a ) , which defines the probability of observing o if the agent has taken action a and is now in s0 . In ACNO-MDPs , however , the observation space is specifically constrained to be O = S [ { missingness } , where the “ missingness ” observation provides no information whatsoever about the underlying state . The “ action-contingent noiselessly observable ” aspect of ACNO-MDPs derives from the fact that p ( o|s0 , aobserve ) = 1 if and only if o = s0 , while p ( o0|s0 , aobserve ) = 0 for all other observations o0 6= s0 . Similarly , p ( o|s0 , anot observe ) = 1 if and only if o = “ missingness ” . While observations in ACNO-MDPs are deterministic , transition dynamics may be stochastic . The probability of transitioning to state s0 after taking action a from state s is given by p ( s0|s , a ) . We let b represent a belief distribution over possible states , subscript t the time step within the episode , and H the episode length . We denote the number of times the state-action pair ( s , a ) is visited across episodes as n ( s , a ) . At each time step , the agent receives expected reward r ( s , a ) , which includes the control-associated reward from executing action a in state s plus an observation cost c ( s , a ) 0 . If the state is not observed , c ( s , a ) = 0 . Thus , for any action a , r ( s , aobserve ) = r ( s , anot observe ) + c ( s , aobserve ) . Observation costs are assumed to be known in advance . Without loss of generality , we assume 0 r ( s , a ) Rmax for all states s and actions a . We assume stationary dynamics , observations , and rewards , but allow for nonstationary policies in light of the fixed horizon length . A t-step policy , ⇡t ( · ) , is a function that maps belief state distributions to distributions over actions . We let V ⇡t ( b0 ) = E ( si , ai ) ⇠⇡t Pt 1 i=0 i P s2S bi ( s ) r ( s , ai ) represent the expected return obtained from starting in b0 and following t-step policy , ⇡t , with discount factor 2 [ 0 , 1 ] . An optimal policy , ⇡⇤ , is a policy for which V ⇡ ⇤ ( b ) = max⇡ V ⇡ ( b ) for all beliefs b in the belief space . The goal of ACNO-MDP learning is to find the policy ⇡ that maximizes V ⇡ ( b0 ) . We evaluate our algorithms using the notion of sample complexity [ 30 , 65 ] , a measure of how much data is needed to learn a near-optimal policy . An algorithm is Probably Approximately Correct ( PAC ) if its sample complexity is polynomial in relevant quantities ( e.g. , S , A , H ) with high probability [ 65 ] . Leveraging the specific observation structure of ACNO-MDPs can improve data efficiency in two ways : ( 1 ) when the agent observes during exploration , it can more rapidly learn an accurate approximate transition model ; and ( 2 ) this improved transition model can be used for better state estimation during POMDP planning , as the underlying state transitions for observed and unobserved states are identical . Faster convergence to an accurate model of the environment and subsequently better planning reduce sample complexity bounds , as we demonstrate both theoretically and empirically .

Summary: This paper concerns the class of POMDPs where the agent may obtain a perfect measurement about the hidden state at a cost. The paper argues such POMDPs, called ACNO-POMDPs, appear in many realistic domains such as healthcare. The paper proposes a probably approximately correct (PAC) learning algorithm for ACNO-POMDPs, and proves a finite sample complexity guarantee for it. The sample complexity guarantee is better than in general POMDPs. Further, the paper proposes an extension of the proposed ideas to a more realistic continuous-state and continuous-action setting. The performance of all proposed methods is empirically quantified in three experimental domains, showing that ACNO-POMDP specific methods provide improvements over general POMDP solvers.