Reinforcement Learning with State Observation Costs in Action-Contingent Noiselessly Observable Markov Decision Processes 

1 Introduction . Reinforcement learning ( RL ) , which involves learning to make sequences of good decisions in stochastic environments , has had many impressive successes over the past decade [ 60 , 68 ] . While many of the more visible successes have occurred in simulated environments , such as Atari video games [ 44 , 58 ] , reinforcement learning has increasingly been used to solve real-world problems [ 10 , 46 , 36 , 41 , 75 , 47 , 37 , 51 , 42 , 34 ] . One key difference between reinforcement learning in simulated vs. real-world environments is that , in most simulated environments , the agent can fully observe the underlying state at each time step with no additional observation costs . In contrast , real-world environments frequently entail state observations that are noisy and/or costly , but optional . Environ- * Equal contribution 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . ments in which state information at each time step can be noisy or missing are more appropriately modeled as partially observable Markov decision processes ( POMDPs ) rather than MDPs [ 29 ] . In this paper , we provide theory and algorithms for a special class of POMDPs in which state information is complete/noiseless when observed , but may be missing at any given time step if the agent chooses not to observe the state . We call this class of POMDPs Action-Contingent Noiselessly Observable MDPs ( ACNO-MDPs ) , which can be useful for capturing a number of important realworld settings , such as : ACNO-MDPs in healthcare . Clinicians in Intensive Care Units ( ICUs ) frequently have to make sequences of treatment decisions under uncertainty for patients at risk . While accurate laboratory tests can inform such decisions , administration of these tests carry a significant cost for the patient and the health system [ 25 ] . These costs , together with the fact that frequent testing may be redundant and wasteful [ 12 , 78 ] , appropriately lead clinicians to refrain from constantly observing the patient state ( i.e. , ordering laboratory tests ) [ 1 , 17 ] . Similarly , other settings – such as glucose monitoring to assist in insulin dosing recommendations [ 18 ] , or white blood cell count monitoring to assist in anti-HIV drug dosing [ 15 ] – have modern tools for accurate observations of biomarkers and could be appropriately modeled by our ACNO-MDP framework . ACNO-MDPs for user-adaptive experiences . Applications on mobile phones and other personal devices can collect information on the user ’ s status , such as location , motion , inter-user contact , and background noise , in order to adaptively suggest features that enhance the user ’ s experience [ 69 , 70 ] . While these sensors can in theory always be kept active , battery consumption due to constant sensing would make such applications less desirable to the user . Modeling this problem as an ACNO-MDP could enable a policy that balances battery usage with personalizing the user ’ s experience . 1 Our contributions . Our main contributions are : • Proposing a Probably Approximately Correct “ Observe-then-Plan ” algorithm for tabular ACNO-MDPs that fully observes while exploring , then employs POMDP planning using learned models . The resulting policy can select when to observe to achieve high expected reward in environments with state observations that are costly but optional . • Providing a finite sample guarantee for the above algorithm in environments with discrete states and actions , such that the number of episodes exhibiting worse than near-optimal behavior is no larger than O ( S 5A2H5Rmax 3◆4 ✏3 ) with ◆ = O ( ln ( SAH/ ✏ ) ) with high probability . This improves over prior work targeted towards more general POMDP problems ( Table 4 ) . • Developing an algorithm that extends these ideas to ACNO-MDPs with continuous state and action spaces . We introduce a novel belief state encoder that can be coupled with many modern deep RL algorithms to solve problems with observation costs , which we demonstrate with several simulation experiments . 2 Related work . Learning in POMDPs without a known model of the environment ( i.e. , with neither a model of state-to-state transition dynamics nor a model of observations conditioned on states ) is significantly more difficult than learning in MDPs [ 49 ] . Thus while there is a rich literature on planning in POMDPs with a known environment model [ 3 , 7 , 13 , 23 , 35 , 39 , 40 , 50 , 53 , 54 , 57 , 59 , 62 , 67 , 77 ] , including in Mixed Observability MDPs ( MOMDPs ) [ 61 ] , active perception frameworks [ 19 , 63 ] , and Non-Observable MDPs ( NOMDPs ) [ 38 ] , fewer approaches exist for RL in POMDPs ( POMDPRL ) . One key difficulty in general POMDP-RL is that , without a known observation model , the agent can never know with certainty which states were actually visited . Approximating a transition model for use in subsequent planning thus becomes infeasible without additional assumptions on the observation model . Many approaches in POMDP-RL , such as the Bayes-Adaptive POMDP ( BAPOMDP ) [ 52 , 56 ] and its variants [ 31 , 32 ] , assume access to an informative prior distribution over the observation model and maintain posterior distributions over the environment model parameters . These algorithms plan using belief distributions over states rather than the states themselves . More recent 1While some of the POMDP-RL methods proposed in this paper may not be energy efficient , recent approaches for compiling and compressing policies into approximately equivalent finite-state controllers could enable energy efficient user-adaptive applications that both sense and change behaviors adaptively [ 20 ] . approaches like Deep Recurrent Q-Networks ( DRQN ) [ 22 , 79 ] and Deep Variational Reinforcement Learning ( DVRL ) [ 43 ] use neural network-based sequence models to learn a condensed representation of the agent ’ s action and observation history and apply modern RL algorithms ( e.g. , DQN [ 44 ] , A2C [ 72 ] ) on this learned representation . These approaches show promising empirical performance but lack theoretical guarantees and do not leverage the specific structure of ACNO-MDPs . Theoretical guarantees on a POMDP-RL algorithm ’ s performance can be beneficial when gathering samples under a suboptimal policy is costly . Though recent works have been able to provide theoretical guarantees in surprisingly broad classes of POMDPs , these works either provide regret guarantees ( not sample complexity guarantees ) [ 24 , 66 ] , or make assumptions about the structure of the observation matrix which do not hold in ACNO-MDPs ( e.g. , that the observation matrix is full rank [ 21 , 2 ] or its singular value is bounded away from 0 [ 27 ] ) . To the best of our knowledge , our work therefore provides sample complexity guarantees for a subclass of POMDPs in which no such prior guarantees existed . Closely related to our work , Klima et al . [ 33 ] consider RL in environments where the state is fully observed , but only intermittently . However , whereas Klima et al . [ 33 ] consider environments in which the agent can not control state observation frequency , we address RL in environments where the agent can decide when to observe , albeit at a cost . Similarly , Yoon et al . [ 74 ] and Chang et al . [ 8 ] use RL methods to learn policies for active sensing/dynamic measurement scheduling . In these works , however , the reward is directly tied to prediction accuracy and agents have no influence on the underlying state beyond observation whereas we focus on agents that can influence the state . Since acceptance , it has come to our attention that earlier work by Zubek and Dietterich [ 80 ] , Zubek et al . [ 81 ] , and Bellinger et al . [ 4 ] proposed RL approaches for learning optimal policies that both modify and observe the state at each time step , as in our problem setup . However , these prior works only provide algorithms for tabular environments and do not provide any theoretical analysis bounding the sample complexity . Related contemporaneous works by Yin et al . [ 73 ] and Bellinger et al . [ 5 ] tackle the problem of RL with observation costs in continuous-state POMDPs and MDPs , respectively : Yin et al . [ 73 ] considers a setting where individual elements of the state vector can be observed at each point in time and Bellinger et al . [ 5 ] performs a more in-depth qualitative analysis of agents ’ measurement patterns throughout training . 3 Problem setting . Action-contingent noiselessly observable Markov decision processes ( ACNO-MDPs ) . A discounted , finite-horizon ACNO-MDP is an environment defined by a tuple M = hS , A , O , p , , r , b , Hi , where S and A are state and action spaces with cardinalities S and A , respectively . The action space A consists of tuples { control actions } ⇥ { observe , not observe } . As in POMDPs , the observation space , O , has cardinality O , and is related to S by the observation/emission function p ( o|s0a ) , which defines the probability of observing o if the agent has taken action a and is now in s0 . In ACNO-MDPs , however , the observation space is specifically constrained to be O = S [ { missingness } , where the “ missingness ” observation provides no information whatsoever about the underlying state . The “ action-contingent noiselessly observable ” aspect of ACNO-MDPs derives from the fact that p ( o|s0 , aobserve ) = 1 if and only if o = s0 , while p ( o0|s0 , aobserve ) = 0 for all other observations o0 6= s0 . Similarly , p ( o|s0 , anot observe ) = 1 if and only if o = “ missingness ” . While observations in ACNO-MDPs are deterministic , transition dynamics may be stochastic . The probability of transitioning to state s0 after taking action a from state s is given by p ( s0|s , a ) . We let b represent a belief distribution over possible states , subscript t the time step within the episode , and H the episode length . We denote the number of times the state-action pair ( s , a ) is visited across episodes as n ( s , a ) . At each time step , the agent receives expected reward r ( s , a ) , which includes the control-associated reward from executing action a in state s plus an observation cost c ( s , a ) 0 . If the state is not observed , c ( s , a ) = 0 . Thus , for any action a , r ( s , aobserve ) = r ( s , anot observe ) + c ( s , aobserve ) . Observation costs are assumed to be known in advance . Without loss of generality , we assume 0 r ( s , a ) Rmax for all states s and actions a . We assume stationary dynamics , observations , and rewards , but allow for nonstationary policies in light of the fixed horizon length . A t-step policy , ⇡t ( · ) , is a function that maps belief state distributions to distributions over actions . We let V ⇡t ( b0 ) = E ( si , ai ) ⇠⇡t Pt 1 i=0 i P s2S bi ( s ) r ( s , ai ) represent the expected return obtained from starting in b0 and following t-step policy , ⇡t , with discount factor 2 [ 0 , 1 ] . An optimal policy , ⇡⇤ , is a policy for which V ⇡ ⇤ ( b ) = max⇡ V ⇡ ( b ) for all beliefs b in the belief space . The goal of ACNO-MDP learning is to find the policy ⇡ that maximizes V ⇡ ( b0 ) . We evaluate our algorithms using the notion of sample complexity [ 30 , 65 ] , a measure of how much data is needed to learn a near-optimal policy . An algorithm is Probably Approximately Correct ( PAC ) if its sample complexity is polynomial in relevant quantities ( e.g. , S , A , H ) with high probability [ 65 ] . Leveraging the specific observation structure of ACNO-MDPs can improve data efficiency in two ways : ( 1 ) when the agent observes during exploration , it can more rapidly learn an accurate approximate transition model ; and ( 2 ) this improved transition model can be used for better state estimation during POMDP planning , as the underlying state transitions for observed and unobserved states are identical . Faster convergence to an accurate model of the environment and subsequently better planning reduce sample complexity bounds , as we demonstrate both theoretically and empirically .

Summary: The paper introduces a new specialisation of POMDPs in which actions are annotated with an option to perform additional sensing. Additional sensing incurs extra cost, but its observations are deterministic, which means that the planning agent receives full information about the underlying hidden state whenever the new sensing action is performed. This POMDP formulation is sensible, and there may be practical applications in which it could be used. The authors derive PAC bounds for planning with this new POMDP specialisation, and propose planning algorithms that can exploit the specific nature of their POMDP definition. Both the theory (i.e. PAC bounds) and the empirical results are encouraging.