Title: Efficient RL with Impaired Observability: Learning to Act with Delayed and Missing State Observations
Venue: NeurIPS 2023
Accepted: yes
Keywords: Delayed and missing observations

MDPs

efficient regret bounds
Abstract: In real-world reinforcement learning (RL) systems, various forms of {\it impaired observability} can complicate matters. These situations arise when an agent is unable to observe the most recent state of the system due to latency or lossy channels, yet the agent must still make real-time decisions. This paper introduces a theoretical investigation into efficient RL in control systems where agents must act with delayed and missing state observations. We establish near-optimal regret bounds, of the form $\tilde{\mathcal{O}}(\sqrt{{\rm poly}(H) SAK})$, for RL in both the delayed and missing observation settings. Despite impaired observability posing significant challenges to the policy class and planning, our results demonstrate that learning remains efficient, with the regret bound optimally depending on the state-action size of the original system. Additionally, we provide a characterization of the performance of the optimal policy under impaired observability, comparing it to the optimal value obtained with full observability.

Efficient RL with Impaired Observability: Learning to Act with Delayed and Missing State Observations

Abstract

In real-world reinforcement learning (RL) systems, various forms of impaired observability can complicate matters. These situations arise when an agent is unable to observe the most recent state of the system due to latency or lossy channels, yet the agent must still make real-time decisions. This paper introduces a theoretical investigation into efficient RL in control systems where agents must act with delayed and missing state observations. We establish near-optimal regret bounds, of the form ˜ O ( √ poly( H ) SAK ) , for RL in both the delayed and missing observation settings. Despite impaired observability posing significant challenges to the policy class and planning, our results demonstrate that learning remains efficient, with the regret bound optimally depending on the state-action size of the original system. Additionally, we provide a characterization of the performance of the optimal policy under impaired observability, comparing it to the optimal value obtained with full observability.

1 Introduction

In Reinforcement Learning (RL), an agent engages with an environment in a sequential manner. In an ideal setting, at each time step, the agent would observe the current state of the environment, select an action to perform, and receive a reward Smallwood and Sondik [1973], Bertsekas [2012], Sutton and Barto [2018], Lattimore and Szepesvári [2020]. However, real-world engineering systems often introduce impaired observability and latency, where the agent may not have immediate access to the instant state and reward information. In systems with lossy communication channels, certain state observations may even be permanently missing, never reaching the agent. Nevertheless, the agent is still required to make real-time decisions based on the available information.

The presence of impaired observability transforms the system into a complex interactive decision process (Figure 1), presenting challenges for both learning and planning in RL. With limited knowledge about recent states and rewards, the agent's policy must extract information from the observed history and utilize it to make immediate decisions. This introduces significant complexity to the policy class and poses difficulties for RL. Moreover, the loss of information due to permanently missing observations further hampers the efficiency of RL methods. Although a naïve approach would involve augmenting the state and action space to create a fully observable Markov Decision Process (MDP), such a method would lead to exponential regret growth in the state-action size.

Why existing methods do not work. One may be tempted to cast the problem of impaired observability into a Partially Observed MDPs (POMDPs). However, this would not solve the problem. In POMDP, the system does not reveal its instant state to the agent but provides an emission state observation conditioned on the latent state. POMDPs are known to suffer from the curse of history Papadimitriou and Tsitsiklis [1987], Bertsekas [2012], Krishnamurthy [2016], unless additional assumptions are imposed. Existing efficient algorithms focus on subclasses of POMDPs with decodable or distinguishable partial observations Jin et al. [2020], Uehara et al. [2022], Zhan et al.

<latexi sh

1_b

64="

SQj

PF8

m

wZvnp

z

9

rXY

>A

C

H

c

V

7T

M

Kq5R

gIGB

J

N

0

fU

uO

2k

d

W

+

o

3

E

/

DL

y

<latexi sh

1_b

64="u mv

RE2

qH

fVo

AI

M

>

C

n

c

DLS

N

F

pr

Zd

B

g

J

w

W

9

Q5

3

y

T

O

/

G

z

7

K

j

P

8

0

YU

+

X

k

<latexi sh

1_b

64="Onp72

8FrYqdmD

MU/

E

k

Q

>A

C

3

c

V

LSg

TX

W+

BIv

y

j

K9

H

Z

u

5

f

o

J

Nz

w

R

G

P

0

<latexi sh

1_b

64="7QJ8

ZdoNBunrA

5mkLM

G

>

C

H

c

VD

S

F

3U

2vqE

g0W

RX

O

T

Ip

Yz/

+

y

9w

P

K

f

j

<latexi sh

1_b

64="

5QP/BNo23F

H

u

n

Yd79

gC

>A

D

c

V

LS

J

U

vW

p

y

r

0I

wmk

Z

+

K

T

f

z

G

8

M

X

O

R

j

q

E

Agent

Environment

Delay

Missing

Policy

s

h

-

d

a

h

-

d

, . . . , a

h

-

1

Time

h

Observation

(

s

h

+1

, r

h

+1

)

π

h

(

·|

s

h

-

d

, a

h

-

d

, . . . , a

h

-

1

)

Real-Time Action

a

h

∼

[2022], Chen et al. [2022], Liu et al. [2022], Zhong et al. [2022], Chen et al. [2023], where the unseen instant state can be inferred from recent observations. Unfortunately, MDPs with impaired observability do not fall into these benign subclasses. The reason behind this is that at each time step, a new observation, if any, is in fact a past state. Viewing it as an emission state of the current one leads to a time reversal posterior distribution depending on the underlying transitions, which suffers from the curse of history and makes the POMDP intractable. The problem becomes even harder if some observations get missing.

Empirical evidences suggested that efficient RL is possible even with impaired state observability Lizotte et al. [2008], Liu et al. [2014], Agarwal and Aggarwal [2021]. However, theoretical understanding of this problem is very limited. One notable work Walsh et al. [2007] studied learning with constant-time delayed observations. They identified subclasses of MDPs with nearly deterministic transitions that can be efficiently learned. Beyond this special case, efficient RL with impaired observability in MDPs with fully generality remains largely open.

Some recent works studied delayed feedback in MDPs Yang et al. [2023], Howson et al. [2023]. It is a fundamentally different problem where the agent's policy can still access real-time states but learning uses delayed data. Our problem is fundamentally harder because the agent's policy can only access the lossy and delayed history. See Section 1 for more discussions.

Our results. In this paper, we provide algorithms and regret analysis for learning the optimal policy in tabular MDPs with impaired observability. Note that this optimal policy is a different one from the optimal policy with full observability. To approach this problem, we construct an augmented MDP reformulation where the original state space is expanded to include available observations of past state and an action sequence. However, the expanded state space is much larger than the original one and naïve application of known methods would lead to exponentially large regret bounds. In our analysis, we exploit structure of the augmented transition model to achieve efficient learning and sharp regret bounds. The main results are summarized as follows.

For MDPs with stochastic delays, we prove a sharp ˜ O ( H 4 √ SAK ) regret bound (Theorem 4.1) comparing to the best feasible policy, Here S and A are the sizes of the original state and action spaces, respectively, H is the horizon, and K is the number of episodes. Here we allows the delay to be stochastic and conditionally independent given on current state and action. Moreover, we quantify the performance degradation of optimal value due to impaired observability, compared to optimal value of fully observable MDPs (Proposition B.2). We also showcase in Proposition 4.2 that a short delay does not reduce the optimal value, but slightly longer delay leads to substantial degradation.

For MDPs with randomly missing observations, we provide an optimistic RL method that provably achieves ˜ O ( √ H 3 S 2 AK ) regret (Proposition 5.1). We also provide a sharper ˜ O ( H 4 √ SAK ) regret in the case when the missing rate is sufficiently small (Theorem 5.2).

To our best knowledge, these results present a first set of theories for RL with delayed and missing observations. Remarkably, our regret bounds nearly match the minimax-optimal regret of standard MDP in their dependence on S, A (noting that the target optimal policies are different in the two cases). It implies that RL with impaired observability are provably as efficient as RL with full observability (up to poly factors of H ).

Related work. Efficient algorithms for learning in the standard setting of tabular MDPs without impaired observability has been extensively studied Kearns and Singh [2002], Brafman and Tennenholtz [2002], Jaksch et al. [2010], Dann and Brunskill [2015], Azar et al. [2017], Agrawal and Jia [2017],

Jin et al. [2018], Dann et al. [2019], Zanette and Brunskill [2019], Zhang et al. [2020], Domingues et al. [2021], where the minimax optimal regret is ˜ O ( √ H 3 SAK ) Azar et al. [2017], Domingues et al. [2021].

The delayed observation studied in this paper is related to delayed feedback in Howson et al. [2023], Yang et al. [2023], yet the setup is fundamentally different. In delayed feedback, an agent sends a policy to the environment for execution. The environment executes the policy on behalf of the agent for an episode, but the whole trajectory will be returned to the agent after some episodes. The policy executed by the environment is able to 'see" instant state and reward. It is Markov and not played by the agent. Our setting concerns learning executable policies when delayed or missing states appear within an episode. The policy is no longer Markov and can only prescribe action based on history. Therefore, the algorithms and analyses for delayed feedback MDPs are not applicable to our settings.

Despite the distinct settings, there are existing fruitful results in efficiently learning MDPs or bandits with delayed feedback. Stochastic delayed feedback in bandits is studied in Agarwal and Duchi [2011], Dudik et al. [2011], Joulani et al. [2013], Vernade et al. [2017, 2020], Gael et al. [2020], Lancewicki et al. [2021]. In the more challenging setting of reinforcement learning, Howson et al. [2023] considers tabular MDPs and Yang et al. [2023] generalizes to MDPs with function approximation and multi-agent settings.

On the other hand, results analyzing MDPs with missing observations are limited in literature, although missing data is a commonly recognized issue in applications García-Laencina et al. [2010], Jerez et al. [2010], Little et al. [2012], Emmanuel et al. [2021]. One notable result is Bouneffouf et al. [2020] for bandits with missing rewards.

Notation : For real numbers a, b , we denote a ∧ b = min { a, b } . In episodic MDPs, we use the superscript k to denote the index of episodes, and the subscript h to denote the index of time. We denote a i : j = { a i , . . . , a j } as the collection of actions from time i to j . For two probability distributions µ and ν , we denote their total variation distance as ∥ µ -ν ∥ TV .

MDP preliminary : An episodic MDP is described by a tuple ( S , A , H, R, P ) , where S , A are state and action spaces, respectively, H is the horizon, R = { r h } H h =1 is the reward function and P = { p h } H h =1 is the transition probability. We primarily focus on tabular MDPs, where S = |S| and A = |A| are both finite. We also assume that the reward is uniformly bounded with ∥ r h ∥ ∞ ≤ 1 for any h . An agent will interact with the environment for K episodes, hoping to find a good policy to maximize the cumulative reward. Within an episode, at the h -th step, the agent chooses an action based on the available information of the environment. After taking the action, the underlying environment produces a reward and transits to the next state. With full state observation, a policy π maps instant state s to an action a or an action distribution. Given such a policy π , the value function is V π h ( s 1 ) = E π [ ∑ H h ′ = h r h ( s h ′ , a h ′ ) ∣ ∣ s h ] , where E π is the policy induced expectation.

2 Problem formulation

In this work, we study MDPs with impaired observability. We focus on two practical settings: 1) delayed observations and 2) missing observations.

2.1 MDPwith delayed observations

In any episode, we denote d h ∈ { 0 , 1 , . . . } as the observational delay of the state and reward at step h . That is, we receive s h and r h at time h + d h . The delay time d h can be dependent on the state s h and action a h at time h . To facilitate analysis, we denote the inter-arrival time between the arrival of observations for step h and h +1 as ∆ h = d h +1 -d h . With delays, at time h , the nearest observable state is denoted as s t h , where t h = argmax { I : ∑ I i =0 ∆ i ≤ h } . Then the executable policy class

chooses actions depending on the nearest visible state and history actions. We impose the following assumption on the interarrival time.

Assumption 2.1 . The interarrival time ∆ h takes value in { 0 , 1 , . . . } . The distribution D h ( s h , a h ) of ∆ h can depend on ( s h , a h ) , but is conditionally independent of the MDP transitions given ( s h , a h ) .

Assumption 2.1 does not impose any specific distributional assumption on ∆ h , but only requires that the delayed observations arrive in order and at each time step, there is at most one new visible state and reward pair ( ∆ h ≥ 0 ). A widely studied example of delays in literature is that the inter-arrival time is geometrically distributed Winsten [1959]. Then the observation sequence { h + d h } is known as a Bernoulli process, which is understood as the discretized version of a Poisson process.

Our delayed observation setting is newly proposed and substantially generalizes the Constant Delayed MDPs(CDMDPs) studied in Brooks and Leondes [1972], Bander and White III [1999], Katsikopoulos and Engelbrecht [2003], Walsh et al. [2007]. When ∆ h = 0 being deterministic for all h ≥ 1 and k , our observation delay coincides with CDMDPs. In CDMDPs, a new past observation is guaranteed to arrive at each time step. However, our delayed model can result in no new observation at some time steps.

Observation delay leads to difficulty in planning, as the agent can only infer the current state and then choose an action. Therefore, the policy is naturally history dependent. We summarize the interaction protocol of the agent with the environment in Protocol 1. At the end of each episode, we can collect

Protocol 1 Interaction between the agent and the environment with delayed observations

1: for episode k = 1 , . . . , K do

2: for time h = 1 , . . . , H do

4: The agent plays action a k h according to some executable policy π k h ∈ Π exec .

3: The agent observes a pair of new, if any, state and reward ( s k t h , a k t h ) . By memory, the agent also has access to past actions a k t h : h -1 .

5: The environment transits to next state s k h +1 ∼ p h ( ·| s k h , a k h ) , which is unobservable to the k k k k

6: end for

agent. The environment also decides the delay at step h +1 as d h +1 = d h +∆ h and t h +1 .

7: The environment sends all unobserved pairs of state and reward as well as their corresponding delay time to the agent.

8: end for

all delayed observations, however, these observations are not used in planning. In reality, the agent can collect these observations by waiting after time H . Protocol 1 is similar to hindsight observability in POMDPs studied in Lee et al. [2023]. Yet their analysis for POMDPs is not directly transferable to our settings as mentioned in the introduction.

2.2 MDPwith missing observations

In addition to the stochastic delay in observations, we also consider randomly missing observations. In applications, an agent interacts with the environment through some communication channel. The communication channel is often imperfect and thus, observation can be lost during transmission. This type of missing is permanent and we describe in the following assumption.

Assumption 2.2 . Any pair of observation (state and reward) is independently observable in the communication channel. The observation rate is λ h depending on h , but independent of the MDP transitions. Moreover, there exists a constant λ 0 such that λ h ≥ λ 0 for any h . The agent will be informed when an observation is missing.

Equivalently, the missing observation rate in Assumption 2.2 is 1 -λ h and assumes the upper bound of 1 -λ 0 . We will show later that this missing rate directly influences the learning efficiency in Section 5.

3 Construction of augmented MDPs

To tackle the limited observability, we expand the original state space and define an augmented MDP. It will serve as the basis for our subsequent theoretical analysis.

3.1 Augmented MDPs with expected reward

In the remainder of this section, we focus on the delayed observation case and defer the missing case to Section 5. Define τ h = { s t h , a t h : h -1 , δ t h } as the augmented state, where δ t h ∈ [0 , ∆ t h ] is the delayed steps after observing ( s t h , r t h ) . Let S aug denote the augmented state space of all possible τ 's. Then the original MDP with delayed observations can be reformulated into a state-augmented one MDP aug = ( S aug , A , H, R aug , P aug ) . The reward is defined as

which is the expected reward given the nearest past state s t h and history actions a t h : h . We can define belief distribution b h ( s | τ h ) = P ( s h = s | τ h ) . Then r h, aug ( τ h , a h ) = E s ∼ b h ( ·| τ h ) [ r ( s, a h )] . Belief distributions are widely adopted in partially observed MDPs Ross et al. [2007], Poupart and Vlassis [2008]. We will frequently use the belief distribution to study the expressivity of Π exec in Section 4.2.

The transition probabilities P aug are sparse. For any τ h = { s t h , a t h : h -1 , δ t h } and τ h +1 = { s t h +1 , a t h +1 : h , δ t h +1 } , we have

where M a ( τ h , τ h +1 ) indicates whether the rolling actions are matched, i.e.,

and θ delay ( s t h , a t h , δ t h ) is defined as

The factored form of θ delay ( s t h , a t h , δ t h ) p t h ( s t h +1 | s t h , a t h ) follows from the conditional independence in Assumption 2.1. We define Q -functions and value functions as follows. For any τ h , a h and policy π ∈ Π exec , we have

We note that V π h is equivalent to V π h, aug for the same executable policy π ∈ Π exec . We also denote P h, aug as the transition operator corresponding to P aug . It can be checked that

MDP aug also appears in makes all the policies in Π exec executable and Markov. Meanwhile, the reward function keeps track of all the expected reward for H steps. Although the expanded state space S aug is much more complicated than the original state space S , the sparse structures in the transition probabilities still allow an efficient exploration. We note that p h, aug only depends on the delay distribution and one-step Markov transitions. However, there is still one caveat for learning in MDP aug - the reward function depends belief distributions, which involve multi-step transitions.

3.2 Augmented MDPs with past reward

To tackle the aforementioned challenge, we further define ˜ MDP aug = ( ˜ S aug , A , ˜ H, ˜ R aug , ˜ P aug ) that shares the optimal policy in MDP aug with an enlonged horizon ˜ H = 2 H . The state space ˜ S aug consists of any τ h = { s t h , a t h : h ∧ H , δ t h } . Comparing to S aug , we cut off the action at horizon H , since a h for h > H has no influence on the state and reward in time [1 , H ] . The reward function is defined as

By definition, ˜ r aug ( τ h , a h ) is a past reward. More importantly, ˜ r h, aug ( τ h , a h ) zeros out rewards outside the original horizon H . Meanwhile, between the arrival of two consecutive state observations, the reward only counts once. Lastly, the transition probabilities are

We interpret the transitions as follows. When h ≤ H , the transition is the same as MDP aug . When h > H , we simply wait for unobserved states and rewards to come. As mentioned, actions taken beyond time H are irrelevant. We build an equivalence in the expected values of MDP aug and ˜ MDP aug .

Proposition 3.1. Let MDP aug and ˜ MDP aug be defined as in the previous paragraphs. Then for any initial state τ 1 and any policy π = { π h } H h =1 ∈ Π exec , it holds that

where in the right-hand side, the policy for steps H +1 to ˜ H is arbitrary.

The proof is provided in Appendix A.1. Proposition 3.1 implies that learning in MDP aug until time H is equivalent to that in ˜ MDP aug for ˜ H steps.

4 RL with delayed observations and regret bound

In this section, we provide regret analysis of learning in MDPs with stochastic delays. For the sake of simplicity, we assume the reward is known, however, extension to unknown reward causes no real difficulty. Motivated by the augmented MDP reformulation, we introduce our learning algorithm in Algorithm 2. In Line 5, unobserved states and rewards are returned to the agent as described in Protocol 1. Using the data set, we construct bonus functions compensating the uncertainty in one-step transitions of the original MDP. This largely sharpens the confidence region, yet still ensures a valid optimism. We emphasize that in Line 9, we are planning on ˜ MDP aug involving the augmented transitions and expanded states of τ ∈ ˜ S aug . Only in this way, we can obtain an executable policy in delayed MDPs. The planning complexity is SA H though.

4.1 Regret bound

We define regret in delayed MDP as

where V π 1 is the value function of the original MDP. Although the regret here is defined on the original MDP, it is equivalent to the regret of the same policy on MDP aug and further ˜ MDP aug by Proposition 3.1. Note that we are comparing with the best executable policy. The performance degradation caused by observation delay is discussed in Section 4.2. The following theorem bounds the regret.

Theorem 4.1 (Regret bound for Delayed MDP) . Suppose Assumption 2.1 holds. Let γ ∈ (0 , 1) be any failure probability. With probaiblity 1 -γ , the regret of Algorithm 2 satisfies

where ι = log SAHK γ and c is a constant.

The proof is provided in Appendix B.1. We discuss several implications.

Algorithm 2 Policy learning for delayed MDPs using ˜ MDP aug

1: Input : Original horizon H , extended horizon ˜ H , policy class Π exec , failure probability γ .

˜ ˜ 3: for episode k = 1 , . . . , K do

2: Init : V H +1 ( τ ) = 0 and Q H ( τ, a ) = H for any τ and a , data set D 0 = ∅ , initial policy π 0 .

4: Execute policy π k -1 for ˜ H steps.

6: On data set D k , compute counting numbers N k h ( s h , a h ) , N k h ( s h , a h , s h +1 ) and N k h ( s h , a h , δ h ) = ∑ k j =1 1 { s j h = s h , a j h = a h , ∆ j h = δ h } .

5: After the episode ends, collect data D k = D k -1 ∪ { ( s k h , a k h , r k h , ∆ k h ) } H h =1 .

7: Estimate transition probabilities and delay distributions via

Then estimators of ˜ p h, aug in ˜ MDP aug is computed using ̂ p k h and ̂ θ k delay .

8: Set bonus function as

for ι = log SAKH γ and c sufficiently large.

9: Run optimistic value iteration in MDP aug for ˜ H steps and obtain π k Π exec

10: end for

˜ ∈ .

11: Return : Learned policy π k 1: H for k = 1 , . . . , K .

Sharp dependence on S and A Theorem 4.1 has a sharp dependence on S and A , although the expanded state space ˜ S aug has a cardinality bounded by SA H . Naïvely learning and planning in ˜ MDP aug would suffer from the exponential enlargement of A H . However, we identify the sparse structures in the transition probabilities. As can be seen, ˜ p h, aug only involves one-step transitions in the original MDP and some conditionally independent delay distributions. Such structures lead to a rather easy estimation of ˜ p h, aug, which can be constructed from the estimators of one-step transitions in the original MDP. Meanwhile, the sparse structures make exploration in ˜ MDP aug efficient, due to many unreachable states.

Effect of the delay distribution and delay length Theorem 4.1 holds for arbitrary conditionally independent delay distributions, even include heavy-tailed distributions. In the worst case of unbounded delays, Theorem 4.1 gives rise to a O ( H 4 √ SAKι ) regret. The reason to this is that if the delay is larger than H , then the corresponding state will only be observed after an episode ends and won't be used in planning. Therefore, we can truncate the delay at H , regardless of its tail distributions.

When the maximal length of delay is bounded by D < H , e.g., CDMDPs with d h = D for any h , Theorem 4.1 implies that the regret is bounded by

for a constant c . Aproof is provided in Appendix B.2. As can be seen, as the length of delay increases, the regret bound enlarges, reflecting the increased difficulty of long delays. Moreover, when D = 0 , that is, no observation delays, the regret bound recovers that in standard MDPs.

4.2 Performance degradation of policy class Π exec

This section devotes to quantify the performance degradation caused by delayed observations. In particular, we bound the value difference between the best executable policy and the best Markov policy in a no delay environment. Recall that V 1 is the value function of the original MDP. We denote

as the best vanilla optimal policy and executable policy, respectively. The values achieved by π ∗ nodelay and π ∗ delay are denoted as V ∗ 1 , nodelay ( s 1 ) and V ∗ 1 , delay ( s 1 ) , respectively. The gap between V ∗ 1 , nodelay and V ∗ 1 , delay quantifies the performance degradation, which is denoted as gap ( s 1 ) = V ∗ 1 , nodelay ( s 1 ) -V ∗ 1 , delay ( s 1 ) . We bound gap in Proposition B.2 in Appendix due to space limit.

In a nutshell, we show that the performance degradation gap is highly relevant to the belief distribution b h ( ·| τ ) . When b h ( ·| τ ) is evenly spread, meaning that the entropy of b h is high and inferring the current unseen state is difficult, we potentially suffer from a large gap . On the contrary, when b h ( ·| τ ) is nearly deterministic, the performance degradation is small. In the special case of deterministic transitions, we have gap = 0 .

4.3 The (mysterious) effect of delay on the optimal value

To further understand the effect of the delay on the optimal value, we provide the following dichotomy. On the one hand, we show that there exists an MDP instance, such that a constant delay of d steps does not hurt the performance. On the other hand, in the same MDP instance, a constant delay of d +1 steps suffers from a constant performance drop.

Proposition 4.2. Consider constant delayed MDPs. Fix a positive integer d < H . Then there exists an MDP instance such that the following two items hold simultaneously.

When delay is d , it holds that 1 K ∑ K k =1 gap ( s k 1 ) = 0 .

When delay is d +1 , it holds that 1 K ∑ K k =1 gap ( s k 1 ) ≥ 1 2 -√ 1 2 K log 1 γ , with probability 1 -γ .

The proof is provided in Appendix B.4. We remark that Proposition 4.2 says that observation delay can be dangerous, even with the slightest possible number of steps. The idea behind Proposition 4.2 is consistent with the analysis on gap . In particular, we construct an MDP instance demonstrated in Figure 2. The reward vanishes at all times but d + 1 . When delay is d , the initial state s 1 is revealed and the policy can choose the best action to receive a reward. When delay is d +1 , however, there is always a 1 / 2 probability of missing the best action for any policy, which leads to a constant performance degradation.

s

1

s

2

· · · · · ·

· · · · · ·

s

1

s

2

s

1

s

2

d

+1

d

+2

s

1

s

2

· · · · · ·

· · · · · ·

s

1

s

1

s

2

s

2

H

Uniform

a

1

a

2

r

d

+1

(

s

i

, a

i

) = 1

for

i

= 1

,

2

5 RL with missing observations and regret analysis

We now switch our study to MDPs with missing observations. In such an environment, executable policies share the same structures as delayed MDPs, where an action is taken based on available history information. Compared to delayed observations, learning with missing observations is more challenging. Since unobserved states and rewards are never revealed, we are suffering from information loss. Besides, we will frequently deal with multi-step transitions, due to missing observations between two consecutive visible states.

5.1 Optimistic planning with missing observations

Despite the difficulty, we present here algorithms that are efficient in learning and planning for MDPs with missing observations. We begin with an optimistic planning algorithm in Algorithm 3. To unify the notation, we denote s k h = ∅ and r k h = ∅ as missing the observation.

<latexi sh

1_b

64="

Q

0+R9

k3rnDB7O

SXz

KGLE

>A

/

c

V

N

F

2p

f

Zdu ovgq

T

m

M

C

wJ

W

j

H

U

P

8

Y

5

I

y

<latexi sh

1_b

64="9yON

n

v

C

MH

Bj

WJ

dgzA

>

/

c

VDLS

F

TX7

+q

I

r2

uK

o

Q

m

Y0kw

R

3

G

P

E

f

U

pZ

8

5

<latexi sh

1_b

64="

IAWu8

coY

+VMEyJ

32

>

B/n

DLS

N

F

pr

f

Zd gq

T

G5

7QP

U

0

kz

j

w

m

O

H

R

X

K

9

<latexi sh

1_b

64="

K

fY

L0E

+oW5

uMS8

pyPCAQ

>

B/n

c

VD

N

F

2

r

Zd vgq

T

7

J

m

wR9

3

z

Gk

H

U

O

I

X

j

<latexi sh

1_b

64="

Q

0+R9

k3rnDB7O

SXz

KGLE

>A

/

c

V

N

F

2p

f

Zdu ovgq

T

m

M

C

wJ

W

j

H

U

P

8

Y

5

I

y

<latexi sh

1_b

64="9yON

n

v

C

MH

Bj

WJ

dgzA

>

/

c

VDLS

F

TX7

+q

I

r2

uK

o

Q

m

Y0kw

R

3

G

P

E

f

U

pZ

8

5

<latexi sh

1_b

64="IV

Z

q

ENJ

F2

A

>

D

S

3Ud

X

o

R

W

p

g5OH

j

v

Tf

m

9

Y

r

y

k

0

P

B

/

w

7

<latexi sh

1_b

64="d oBmwq

HTk+v9V

R

>A

c

DLSgM

F

3U

2

K

jP

7

Y

J

p

N

G

8

/

E

0Q

nXu

I

z

r

5

O

W

<latexi sh

1_b

64="d oBmwq

HTk+v9V

R

>A

c

DLSgM

F

3U

2

K

jP

7

Y

J

p

N

G

8

/

E

0Q

nXu

I

z

r

5

O

W

<latexi sh

1_b

64="qr

8U

TJ0

Z

Y

F

RdDV9

>A

B/n

c

LS

N

2p

f

u

o

gC

X

G5

Q

M

3

y

I

v

W

+

EH

7

O

m

j

z

P

w

k

K

<latexi sh

1_b

64="

Q

0+R9

k3rnDB7O

SXz

KGLE

>A

/

c

V

N

F

2p

f

Zdu ovgq

T

m

M

C

wJ

W

j

H

U

P

8

Y

5

I

y

<latexi sh

1_b

64="9yON

n

v

C

MH

Bj

WJ

dgzA

>

/

c

VDLS

F

TX7

+q

I

r2

uK

o

Q

m

Y0kw

R

3

G

P

E

f

U

pZ

8

5

<latexi sh

1_b

64="dMk

5O

0uT

g2

+HD3

q

/ZoI

>A

B

n

c

V

LS

N

F

pr

f

C

U

9

E

W

ym

R

J

w

K

P

v

8

Y

X

j

7

zQ

G

<latexi sh

1_b

64="

Q

0+R9

k3rnDB7O

SXz

KGLE

>A

/

c

V

N

F

2p

f

Zdu ovgq

T

m

M

C

wJ

W

j

H

U

P

8

Y

5

I

y

<latexi sh

1_b

64="9yON

n

v

C

MH

Bj

WJ

dgzA

>

/

c

VDLS

F

TX7

+q

I

r2

uK

o

Q

m

Y0kw

R

3

G

P

E

f

U

pZ

8

5

<latexi sh

1_b

64="

Q

0+R9

k3rnDB7O

SXz

KGLE

>A

/

c

V

N

F

2p

f

Zdu ovgq

T

m

M

C

wJ

W

j

H

U

P

8

Y

5

I

y

<latexi sh

1_b

64="9yON

n

v

C

MH

Bj

WJ

dgzA

>

/

c

VDLS

F

TX7

+q

I

r2

uK

o

Q

m

Y0kw

R

3

G

P

E

f

U

pZ

8

5

<latexi sh

1_b

64="d oBmwq

HTk+v9V

R

>A

c

DLSgM

F

3U

2

K

jP

7

Y

J

p

N

G

8

/

E

0Q

nXu

I

z

r

5

O

W

<latexi sh

1_b

64="d oBmwq

HTk+v9V

R

>A

c

DLSgM

F

3U

2

K

jP

7

Y

J

p

N

G

8

/

E

0Q

nXu

I

z

r

5

O

W

<latexi sh

1_b

64="0z

ZrQJ

Xc

A

v

UmkIS

Y

>

B/H

VD

gN

F

wT

3qE

j

G

7R

MC

2d

n

u

8

9

+

L

f

O

P

Wy

5

p

<latexi sh

1_b

64="

Q

0+R9

k3rnDB7O

SXz

KGLE

>A

/

c

V

N

F

2p

f

Zdu ovgq

T

m

M

C

wJ

W

j

H

U

P

8

Y

5

I

y

<latexi sh

1_b

64="9yON

n

v

C

MH

Bj

WJ

dgzA

>

/

c

VDLS

F

TX7

+q

I

r2

uK

o

Q

m

Y0kw

R

3

G

P

E

f

U

pZ

8

5

Algorithm 3 Optimistic planning for MDPs with missing observations

1: Input : Horizon H , observable rate λ h .

3: for episode k = 1 , . . . , K do

2: Init : B 0 = Θ to be all possible tabular MDPs, data set D 0 = ∅ .

4: Set policy π k = argmax π ∈ Π exec max θ ∈B k V π 1 ,θ ( s k 1 ) .

6: Compute counting number N k h ( s, a ) = ∑ k j =1 1 { s j h = s, a j h = a, s j h +1 = ∅} .

̸

5: Play policy π k and collect data D k -1 ∪ { ( s k h , a k h , r k h ) } H h =1 .

7: Update confidence set

where ̂ p k h ( s ′ | s, a ) = N k h ( s,a,s ′ ) N k h ( s,a ) and c is some constant.

8: end for

The majority of the algorithm resembles the typical optimistic planning Jaksch et al. [2010] but with some notable differences. In Line 4, the value function V 1 ,θ is for the original MDP with transition probabilities parameterized by θ . Different from the typical optimistic planning, the underlying MDP here obeys the stochastic observable model in Assumption 2.2. Therefore, the value V 1 ,θ is the sum of all possible values under missing observations. When counting N k h ( s, a ) in Line 6, we exclude data tuples missing the next state, which inevitably slows down the learning curve. Nonetheless, the effect of missing only contributes as a scaling factor in the regret.

Proposition 5.1. Suppose Assumption 2.2 holds with λ h known. Given a failure probability γ , with probability 1 -γ , the regret of Algorithm 4 satisfies

where ι = log SAHK γ and c is a constant.

The proof is provided in Appendix C.1. Proposition 5.1 is optimal in the K dependence and achieves an S 2 A dependence on the complexity of the underlying MDP. In the extreme case of λ 0 ≈ 0 , which implies that every state and reward are hardly observable, we have Regret ( K ) = ˜ O ( 1 λ 2 0 √ H 3 S 2 AK ) . Here λ 2 0 is the probability of observing two consecutive states for estimating the transition probabilities. Proposition 5.1 requires knowledge of observable rate λ h . This is not a restrictive condition, as estimating λ h from Bernoulli random variables is much easier than estimating transition probabilities.

5.2 Model-based planning using augmented MDPs

Proposition 5.1 has a lenient dependence on the missing rate 1 -λ 2 0 , nonetheless, is not sharp on the dependence of S . We next show that the augmented MDP approach is effective to tackle missing observations, when the observable rate satisfies additional conditions. Specifically, we assume that the observable rate λ h is independent of ( s, a ) . We utilize the MDP aug reformulation, except that we redefine the transition probabilities as

The first case in p h, aug corresponds to receiving the state observation at time h +1 . In contrast to the delayed MDPs, the transition probabilities here potentially rely on multi-step transitions in the original MDP. The second case of the transition corresponds to missing the observation. We summarize the policy learning procedure in Algorithm 4 in Appendix C.2, which is similar to Algorithm 2, but with a new bonus function. The following theorem shows that Algorithm 4 is asymptotically efficient when the observable rate is relatively high.

Theorem 5.2. Suppose Assumption 2.2 holds with λ 0 ≥ 1 -A -(1+ v ) for some positive constant v . Given a failure probability γ , with probability 1 -γ , the regret of Algorithm 4 satisfies

where ι = log SAHK γ and c is a constant.

The proof is provided in Appendix C.2. Some remarks are in order.

SA rate when K is large When the number of episodes K ≥ S 3(1+ v ) /v , the first term H 4 √ SAKι 3 in the regret bound dominates and attains a sharp dependence on S and A . However, when the number of episodes are limited, the regret bound has a worse dependence on the state space size S . We also observe that as the missing rate λ becomes small (equivalently, v becomes large), the regret is close to ˜ O ( H 4 √ SAKι 3 ) .

Observable rate smaller than 1 -1 /A Theorem 5.2 holds for an observable rate λ 0 > 1 -1 /A . The intuition behind is that to fully explore all the actions when a state observation is missing takes A trials. Therefore, in expectation, we will encounter a missing observation at least every A episodes as long as λ 0 > 1 -1 /A . Nonetheless, when λ 0 ≤ 1 -1 /A , the regret bound remains curiously underexplored. We conjecture that λ 0 = 1 -1 /A is a critical point distinguishes unique strategies for learning and planning in MDPs with missing observations. A detailed analysis goes beyond the scope of the current paper.

Proof sketch The proof of Theorem 5.2 adapts the analysis of model-based UCBVI algorithms Azar et al. [2017]. Let m denote the maximal length of consecutive missing observations. We denote E m as the event when the maximal length of consecutive missing is less than m . On event E m , a naïve analysis leads to a ˜ O ( √ poly( H ) SA m +1 K ) regret, in observation to the size of the expanded state space S aug . However, our analysis circumvents the A m dependence by exploiting the occurrence of consecutive missing observations is rare (Lemma C.3). On the complement of event, the regret is bounded by KH (1 -P ( E m )) . Summing up the two parts and choosing a proper m yield our result.

6 Conclusion

In this paper, we have studied learning and planning in impaired observability MDPs. We focus on MDPs with delayed and missing observations. Specifically, for delayed observations, we have shown an efficient ˜ O ( H 4 √ SAK ) regret. For missing observations, we have provided an optimistic planning algorithm achieving an ˜ O ( √ H 3 S 2 AK ) regret. If the missing rate is relatively small, we have shown an efficient ˜ O ( H 4 √ SAK ) regret bound. Further, we have characterized the performance drop caused by impaired observability compared to full observability.

References

Alekh Agarwal and John C Duchi. Distributed delayed stochastic optimization. Advances in Neural Information Processing Systems , 24, 2011.

Mridul Agarwal and Vaneet Aggarwal. Blind decision making: Reinforcement learning with delayed observations. Pattern Recognition Letters , 150:176-182, 2021.

Shipra Agrawal and Randy Jia. Optimistic posterior sampling for reinforcement learning: worst-case regret bounds. Advances in Neural Information Processing Systems , 30, 2017.

Mohammad Gheshlaghi Azar, Ian Osband, and Rémi Munos. Minimax regret bounds for reinforcement learning. In Proceedings of the International Conference on Machine Learning , pages 263-272. PMLR, 2017.

James L Bander and Chelsea C White III. Markov decision processes with noise-corrupted and delayed state observations. Journal of the Operational Research Society , 50(6):660-668, 1999.

Dimitri Bertsekas. Dynamic Programming and Optimal Control: Volume I , volume 1. Athena Scientific, 2012.

Djallel Bouneffouf, Sohini Upadhyay, and Yasaman Khazaeni. Contextual bandit with missing rewards. arXiv preprint arXiv:2007.06368, 2020.

Ronen I Brafman and Moshe Tennenholtz. R-max-a general polynomial time algorithm for nearoptimal reinforcement learning. Journal of Machine Learning Research , 3(Oct):213-231, 2002.

DM Brooks and Cornelius T Leondes. Markov decision processes with state-information lag. Operations Research , 20(4):904-907, 1972.

Fan Chen, Yu Bai, and Song Mei. Partially observable RL with B-stability: Unified structural condition and sharp sample-efficient algorithms. arXiv preprint arXiv:2209.14990, 2022.

Fan Chen, Huan Wang, Caiming Xiong, Song Mei, and Yu Bai. Lower bounds for learning in revealing POMDPs. arXiv preprint arXiv:2302.01333, 2023.

Christoph Dann and Emma Brunskill. Sample complexity of episodic fixed-horizon reinforcement learning. Advances in Neural Information Processing Systems , 28, 2015.

Christoph Dann, Lihong Li, Wei Wei, and Emma Brunskill. Policy certificates: Towards accountable reinforcement learning. In Proceedings of the International Conference on Machine Learning , pages 1507-1516. PMLR, 2019.

Omar Darwiche Domingues, Pierre Ménard, Emilie Kaufmann, and Michal Valko. Episodic reinforcement learning in finite mdps: Minimax lower bounds revisited. In Algorithmic Learning Theory , pages 578-598. PMLR, 2021.

Miroslav Dudik, Daniel Hsu, Satyen Kale, Nikos Karampatziakis, John Langford, Lev Reyzin, and Tong Zhang. Efficient optimal learning for contextual bandits. arXiv preprint arXiv:1106.2369, 2011.

Tlamelo Emmanuel, Thabiso Maupong, Dimane Mpoeleng, Thabo Semong, Banyatsang Mphago, and Oteng Tabona. A survey on missing data in machine learning. Journal of Big Data , 8(1):1-37, 2021.

Manegueu Anne Gael, Claire Vernade, Alexandra Carpentier, and Michal Valko. Stochastic bandits with arm-dependent delays. In Proceedings of the International Conference on Machine Learning , pages 3348-3356. PMLR, 2020.

Pedro J García-Laencina, José-Luis Sancho-Gómez, and Aníbal R Figueiras-Vidal. Pattern classification with missing data: a review. Neural Computing and Applications , 19:263-282, 2010.

Benjamin Howson, Ciara Pike-Burke, and Sarah Filippi. Delayed feedback in generalised linear bandits revisited. In International Conference on Artificial Intelligence and Statistics , pages 6095-6119. PMLR, 2023.

Thomas Jaksch, Ronald Ortner, and Peter Auer. Near-optimal regret bounds for reinforcement learning. Journal of Machine Learning Research , 11(51):1563-1600, 2010. URL http://jmlr. org/papers/v11/jaksch10a.html .

José M Jerez, Ignacio Molina, Pedro J García-Laencina, Emilio Alba, Nuria Ribelles, Miguel Martín, and Leonardo Franco. Missing data imputation using statistical and machine learning methods in a real breast cancer problem. Artificial Intelligence in Medicine , 50(2):105-115, 2010.

Chi Jin, Zeyuan Allen-Zhu, Sebastien Bubeck, and Michael I Jordan. Is Q-learning provably efficient? Advances in Neural Information Processing Systems , 31, 2018.

Chi Jin, Sham Kakade, Akshay Krishnamurthy, and Qinghua Liu. Sample-efficient reinforcement learning of undercomplete POMDPs. Advances in Neural Information Processing Systems , 33: 18530-18539, 2020.

Pooria Joulani, Andras Gyorgy, and Csaba Szepesvári. Online learning under delayed feedback. In Proceedings of the International Conference on Machine Learning , pages 1453-1461. PMLR, 2013.

Konstantinos V Katsikopoulos and Sascha E Engelbrecht. Markov decision processes with delays and asynchronous cost collection. IEEE Transactions on Automatic Control , 48(4):568-574, 2003.

Michael Kearns and Satinder Singh. Near-optimal reinforcement learning in polynomial time. Machine Learning , 49(2):209-232, 2002.

Vikram Krishnamurthy. Partially Observed Markov Decision Processes . Cambridge University Press, 2016.

Tal Lancewicki, Shahar Segal, Tomer Koren, and Yishay Mansour. Stochastic multi-armed bandits with unrestricted delay distributions. In Proceedings of the International Conference on Machine Learning , pages 5969-5978. PMLR, 2021.

Tor Lattimore and Csaba Szepesvári. Bandit Algorithms . Cambridge University Press, 2020.

Jonathan N Lee, Alekh Agarwal, Christoph Dann, and Tong Zhang. Learning in pomdps is sampleefficient with hindsight observability. arXiv preprint arXiv:2301.13857, 2023.

Roderick J Little, Ralph D'Agostino, Michael L Cohen, Kay Dickersin, Scott S Emerson, John T Farrar, Constantine Frangakis, Joseph W Hogan, Geert Molenberghs, Susan A Murphy, et al. The prevention and treatment of missing data in clinical trials. New England Journal of Medicine , 367 (14):1355-1360, 2012.

Qinghua Liu, Praneeth Netrapalli, Csaba Szepesvari, and Chi Jin. Optimistic MLE-A generic model-based algorithm for partially observable sequential decision making. arXiv preprint arXiv:2209.14997, 2022.

Shichao Liu, Xiaoyu Wang, and Peter Xiaoping Liu. Impact of communication delays on secondary frequency control in an islanded microgrid. IEEE Transactions on Industrial Electronics , 62(4): 2021-2031, 2014.

Daniel J Lizotte, Lacey Gunter, Eric Laber, and Susan A Murphy. Missing data and uncertainty in batch reinforcement learning. In Advances in Neural Information Processing Systems , 2008.

Christos H Papadimitriou and John N Tsitsiklis. The complexity of markov decision processes. Mathematics of Operations Research , 12(3):441-450, 1987.

Pascal Poupart and Nikos Vlassis. Model-based bayesian reinforcement learning in partially observable domains. In Proceedings of the International Symposium on Artificial Intelligence and Mathematics , pages 1-2, 2008.

Stephane Ross, Brahim Chaib-draa, and Joelle Pineau. Bayes-adaptive pomdps. Advances in Neural Information Processing Systems , 20, 2007.

Richard D Smallwood and Edward J Sondik. The optimal control of partially observable markov processes over a finite horizon. Operations Research , 21(5):1071-1088, 1973.

Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction . MIT Press, 2018.

Masatoshi Uehara, Ayush Sekhari, Jason D Lee, Nathan Kallus, and Wen Sun. Provably efficient reinforcement learning in partially observable dynamical systems. arXiv preprint arXiv:2206.12020, 2022.

Claire Vernade, Olivier Cappé, and Vianney Perchet. Stochastic bandit models for delayed conversions. arXiv preprint arXiv:1706.09186, 2017.

Claire Vernade, Alexandra Carpentier, Tor Lattimore, Giovanni Zappella, Beyza Ermis, and Michael Brueckner. Linear bandits with stochastic delayed feedback. In Proceedings of the International Conference on Machine Learning , pages 9712-9721. PMLR, 2020.

Martin J Wainwright. High-dimensional Statistics: A Non-asymptotic Viewpoint , volume 48. Cambridge University Press, 2019.

Thomas J Walsh, Ali Nouri, Lihong Li, and Michael L Littman. Planning and learning in environments with delayed feedback. In Proceedings of Machine Learning: ECML 2007: 18th European Conference on Machine Learning, Warsaw, Poland, September 17-21, 2007, pages 442-453. Springer, 2007.

Tsachy Weissman, Erik Ordentlich, Gadiel Seroussi, Sergio Verdu, and Marcelo J Weinberger. Inequalities for the l1 deviation of the empirical distribution. Hewlett-Packard Labs, Technical Report , 2003.

CB Winsten. Geometric distributions in the theory of queues. Journal of the Royal Statistical Society: Series B (Methodological) , 21(1):1-22, 1959.

Yunchang Yang, Han Zhong, Tianhao Wu, Bin Liu, Liwei Wang, and Simon S Du. A reduction-based framework for sequential decision making with delayed feedback. arXiv preprint arXiv:2302.01477, 2023.

Andrea Zanette and Emma Brunskill. Tighter problem-dependent regret bounds in reinforcement learning without domain knowledge using value function bounds. In Proceedings of the International Conference on Machine Learning , pages 7304-7312. PMLR, 2019.

Wenhao Zhan, Masatoshi Uehara, Wen Sun, and Jason D Lee. PAC reinforcement learning for predictive state representations. arXiv preprint arXiv:2207.05738, 2022.

Zihan Zhang, Yuan Zhou, and Xiangyang Ji. Almost optimal model-free reinforcement learning via reference-advantage decomposition. Advances in Neural Information Processing Systems , 33: 15198-15207, 2020.

Han Zhong, Wei Xiong, Sirui Zheng, Liwei Wang, Zhaoran Wang, Zhuoran Yang, and Tong Zhang. A posterior sampling framework for interactive decision making. arXiv preprint arXiv:2211.01962, 2022.

A Omitted proof in Section 3

A.1 Proof of Proposition 3.1

Proof. Consider an arbitrary fixed inter-arrival pattern ∆ 0 , ∆ 1 , . . . , ∆ H -1 . We show that the expected accumulated rewards under this inter-arrival pattern are identical for MDP aug and ˜ MDP aug . In ˜ MDP aug , we have

where equality ( i ) invokes the definition of ˜ r h, aug and equality ( ii ) eliminates zero reward terms. Now taking expectation over all possible inter-arrival patterns, we deduce

The proof is complete.

B Omitted proofs in Section 4

B.1 Proof of Theorem 4.1

Proof. We adapt the main steps from Azar et al. [2017] for proving the theorem. The proof consists of verifying a valid optimism and developing a regret analysis. We denote ˜ Q ∗ h, aug as the optimal Q -function for ˜ MDP aug . When analyzing the regret, we also denote ˜ Q k h, aug as the optimal Q -function in the k -th episode.

Valid optimism To begin with, we verify that the choice of the bonus functions leads to a valid optimism in the following lemma.

Lemma B.1. Given any failure probability γ < 1 , we set a bonus as

where ι = log ( SAHK γ ) and c A is a constant. Then with probability 1 -γ , it holds

Proof of Lemma B.1. We compute the cardinality of the expanded state space ˜ S aug as

For a fixed episode k , we show by backward induction that the assertion in Lemma B.1 holds. To ease the presentation, we omit all superscripts k , all subscripts 'aug', as well as the tilde ˜ · notation.

When h = ˜ H +1 , the base assertion holds immediately. Suppose the assertion is true for time h +1 . At time h , for any fixed ( τ h , a h ) , if Q h ( τ h , a h ) = H , the assertion holds true. Otherwise, we have

We show a lower bound on ( A ) . If h ≥ H , expanding the transition kernel P h leads to

where equality ( i ) requires τ h +1 to take s t h +1 as the new state observation, and inequality ( ii ) follows from the Hoeffding's inequality (Lemma D.2) with a constant c A, 1 . Note that the Hι term in the numerator comes from a union bound over ˜ S aug ×A .

On the other hand, if h < H , expanding the transition kernel P h yields

Note that ( A 1 ) accounts for receiving a new state observation in τ h +1 , and ( A 2 ) accounts for no new state observation. We tackle these two terms separately. For ( A 1 ) , we have

where in ( i ) , the first term is the estimation error of ̂ θ using the collected data, the second term follows from Hoeffding's inequality, and c A, 2 is an absolute constant. For ( A 2 ) , we have

since τ h +1 is now uniquely determined. Summing up ( A 1 ) and ( A 2 ) , we obtain

It remains to bound the estimation error of ̂ θ t h ( s t h , a t h , δ t h ) . Using the Hoeffding's inequality again, we obtain

Taking c A = max { c A, 1 , c A, 2 , c θ , 2 } , we have

With the choice of the bonus function, it can be checked that

with probability 1 -γ for any ( τ h , a h ) .

Regret analysis In the sequel, we omit subscripts 'aug' and tilde ˜ · for simplicity. Thanks to Lemma B.1, we consider ( Q k h -Q π k h ) ( τ k h , a k h ) as an upper bound of ( Q ∗ h -Q π k h ) ( τ k h , a k h ) . We bound ( Q k h -Q π k h ) ( τ k h , a k h ) by

Similar to Lemma B.1, for h ≥ H , we expand term ( A ) into

On the other hand, for h ≤ H , the decomposition of term ( A ) is more complicated. We have

Term ( A 2 ) can be directly bounded by

with probability 1 -γ . To bound ( A 1 ) , we have

Putting ( A 1 ) and ( A 2 ) together, we obtain

In both (B.2) and (B.3) for different ranges of h , we apply the Bernstein inequality (Lemma D.1) to derive

where inequality ( i ) follows from √ ab ≤ a + b . Substituting (B.4) into (B.2), for h ≥ H , we deduce where c ′ is a sufficiently large constant. By the same reasoning, substituting (B.4) into (B.3), for h < H , we have

We denote ζ k h = c ′ SH 2 ι N k t h ( s k t h ,a k t h ) . Now we have a unified upper bound on ( A ) for any h ∈ [1 , ˜ H ] as

Substituting (B.5) back into (B.1), we have

We further denote ξ k h = ( P h [ V k h -V π k h ]) ( τ k h , a k h ) -[ V k h +1 -V π k h +1 ] ( τ k h +1 ) and rewrite ( V k h -V π k h ) ( τ k h ) as

Recall ˜ H = 2 H . Using a recursive summation argument, we deduce

As a consequence, the total regret is bounded by

We need to sum over ζ k h , ξ k h , b k h . Consider ζ k h first. We have

where inequality ( i ) invokes the fact that t h only takes value in { 1 , . . . , H } and each N k t h ( s k t h , a k t h ) is repeated at most H times, and inequality ( ii ) follows from the pigeon-hole argument in Azar et al. [2017].

Next we bound the summation over ξ k h . This is a martingale difference sequence. We apply AzumaHoeffding's inequality (Lemma D.3) with n = 2 H and c i = 4 H to obtain

The additional H dependence above comes from a union bound over ˜ S aug ×A . Lastly, we tackle the summation over bonus functions b k h . We have

Putting (B.7), (B.8) and (B.9) together, we deduce

for some constant c . To this end, the only remaining task is to find ∑ K k =1 ∑ 2 H h =1 √ 1 N k t h ( s k t h ,a k t h ,δ k t h ) , which undergoes a similar argument as the bonus summation. We have

where inequality ( i ) invokes ∑ n i =1 1 / √ i ≤ 2 √ n , inequality ( ii ) follows from Cauchy-Schwarz, and inequality ( iii ) uses the fact that δ is bounded by H . Plugging (B.10) into the regret bound, we obtain the desired result

with probability 1 -γ . Absorbing √ H 4 Kι into H 4 √ SAKι yields the bound in Theorem 4.1.

B.2 Regret under bounded delay

To obtain the regret bound under bounded delay, we only need to modify several steps in the proof of Theorem 4.1. Specifically, in Lemma B.1, we replace H by D +1 in the square root, so that the bonus function becomes

The optimism still holds, since the expanded state space contains at most D historical actions.

The second modification is to observe that since the delay is bounded by D , each counting number N k h only repeats at most D +1 times. In this way, (B.7) becomes

(B.8) and (B.9) are replaced by

respectively. Lastly, we also have

Putting these updated upper bounds together and substituting into Regret ( K ) , we deduce

B.3 Statement and proof of Proposition B.2

Proposition B.2. In the setup of Section 4.2, we have

where ρ π ∗ nodelay h and ρ π ∗ delay h are visitation measures induced by π ∗ nodelay and π ∗ delay , respectively.

Term E 1 is strictly larger than zero due to the convexity of the max operation. Term E 2 accounts for the difference in the visitation measure. When the original MDP has deterministic transitions, we can check that E 1 is zero, since the expectation over s is concentrated on a singleton that can be inferred from history. Hence, the visitation measures are also identical, which implies V ∗ 1 , nodelay ( s 1 ) -V ∗ 1 , delay ( s 1 ) = 0 . On the contrary, when b h ( ·| τ ) is evenly spread, meaning that the entropy of b h is high, we potentially suffer from a large performance drop, in that, inferring the current state is difficult.

Proof of Proposition B.2. Let τ 1 , . . . , τ H denote the states observed in the delayed environment. Since π ∗ nodelay is greedy and Markov, we obtain

Recursively applying the above argument, we deduce

We also rewrite V ∗ 1 , delay ( s 1 ) as

Then we write the difference between V ∗ 1 , nodelay ( s 1 ) and V ∗ 1 , delay ( s 1 ) as

We also have

Combining the above two inequalities, we obtain

The proof is complete.

B.4 Proof of Proposition 4.2

Proof. We construct an MDP instance ( S , A , H, R, P ) for H > d as follows. Let S = { 1 , 2 } and A = { a 1 , a 2 } . For the reward function, we have

The reward is nonzero only at time d +1 . The transition probabilities are defined as

̸

The transition probability at step d +1 says that s ′ is uniform regardless of the previous state and action. Suppose a uniform initial distribution on s 1 . We first show that if the constant delay equals d , then there exists a policy π ∗ ,d achieving maximal value. Indeed, the policy is chosen as

̸

It is straightforward to check that π ∗ ,d is optimal, since at step d +1 , s 1 is revealed and the policy takes the optimal action a s 1 to obtain reward 1 .

On the other hand, if the constant delay equals d + 1 , then any policy suffers from a constant performance degradation. To see this, in a single trajectory, since the starting state is only revealed at time d +2 , the policy at time d +1 cannot exploit the information of the initial state. Therefore, any policy coincides with the best action with probability 1 2 . For K episodes, with probability 1 -γ , the total reward of any policy π ∈ Π exec is bounded by

due to Hoeffding's inequality. As a result, the performance drop is at least

C Omitted proofs in Section 5

C.1 Proof of Proposition 5.1

Proof. We first show that the ground-truth transition probabilities p θ ∗ h belongs to B k with high probability. By Theorem 2.2 in Weissman et al. [2003] (see also Equation (44) in Jaksch et al. [2010]), at the k -th episode, for any fixed ( s, a, h ) , we have

Setting t = c √ Sι N k h ( s,a ) for some constant c ensures that

holds over any ( s, a, h, k ) with probability 1 -γ . As a consequence, the event p θ ∗ h ( ·| s, a ) ∈ B k holds with probability 1 -γ over all ( s, a, h, k ) .

Conditioned on the high probability event p θ ∗ h ∈ B k for all ( h, s, a ) , we have by standard performance difference arguments that

where inequality ( i ) follows from the valid optimism since Line 4 in Algorithm 3 is taken over double maximization, equality ( ii ) recursively expands the value function and ⟨· , ·⟩ denotes the inner product, inequality ( iii ) invokes Azuma-Hoeffding's inequality, and inequality ( iv ) invokes Lemma C.2.

C.2 Algorithm and proof of Theorem 5.2

Algorithm 4 Policy learning for MDPs with missing observations

1: Input : Horizon H .

3: for episode k = 1 , . . . , K do

2: Init : V H +1 ( τ ) = 0 and Q H ( τ, a ) = H for any τ, a , data set D 0 = ∅ , initial policy π 0 .

4: Execute policy π k -1 .

6: On data set D k , compute counting numbers

5: After the episode ends, collect data D k = D k -1 ∪ { ( s k h , a k h , r k h ) } H h =1 .

̸

7: Estimate transition probabilities and delay distributions via

8: Set bonus function as

for ι = log SAKH γ and c sufficiently large.

9: Run optimistic value iteration in MDP aug for H steps and obtain π k ∈ Π exec .

11: Return : Learned policy π k for k = 1 , . . . , K .

10: end for

We remark that similar to delayed MDPs, in Line 9 the planning is on MDP aug and the obtained policy is executable given any τ ∈ S aug when state observation is missed. Therefore, the planning complexity is SA H . Different from Algorithm 2, the bonus function here depends on multi-step transitions, in that missing observations are permanently lost.

Proof of Theorem 5.2. The proof utilizes similar steps as Theorem 4.1, with an extra care on the summation of bonus functions.

Valid optimism We verify the choice of bonus functions leads to a valid optimism.

Lemma C.1. Given any failure probability γ < 1 , we set bonus functions as

Then with probability 1 -γ , it holds

Proof of Lemma C.1. In the proof, we omit subscript 'aug' for simplicity. We use backward induction on time h again. The base case of H + 1 holds immediately due to the initial value of V H +1 , aug . Suppose at time h + 1 , the assertion holds. Then for time h , if Q h, aug = H , the assertion holds trivially. Otherwise, we have

We lower bound ( A ) and ( B ) separately. For term ( A ) , we have

In ( A 1 ) , τ h +1 is { s h +1 } . We bound ( A 1 ) as

where inequality ( i ) invokes Hoeffding's inequality and holds with probability 1 -γ for any τ h , a h and some constant c A . Term ( A 2 ) is immediately bounded by

Putting ( A 1 ) and ( A 2 ) together, we derive

with high probabilty. For term ( B ) , we have

Taking c = c A + c B and summing up ( A ) and ( B ) , we have

We estimate λ h by its empirical average. In episode k ≥ 1 , we have access to k i.i.d. realizations of Bernoulli random variable with rate λ h (observable or not). Therefore, by Hoeffding's inequality, we have

Substituting into Q k h ( τ h , a h ) -Q ∗ h ( τ h , a h ) and reloading constant c give rise to

The proof is complete.

Regret analysis We omit subscript 'aug' to ease the presentation. The same derivation in the proof of Theorem 4.1 gives rise to

Lemma C.1 shows that ( A ) can be written as

Following the derivation in (B.4), (B.5) and (B.6), we have

where ξ k h = ( P h [ V k h -V π k h ]) ( τ k h , a k h ) -[ V k h +1 -V π k h +1 ] ( τ k h +1 ) is the martingale difference and ζ k h = c ′ SH 2 ι N k h ( τ k h ,a k h ) .

Counting number summation The summation over ξ k h is standard. Using the Azuma-Hoeffding's inequality, we have

It remains to find the summations involving N k h ( τ k h , a k h ) . First, we show that the event E m = { h -t h -1 ≤ m } , i.e., the maximal consecutive delay is upper bounded by m> 0 , holds with high probability. We have

since λ 0 is a uniform lower bound of λ h . Next, we provide an upper bound on N K h ( τ h , a h ) . For a given tuple ( h, τ h , a h , t h ) , the consecutive missing length is h -t h -1 . Such a missing pattern appears with probability at most (1 -λ 0 ) h -t h -1 . As a consequence, denote C K h -t h -1 as the number of h -t h -1 consecutive missings in K episodes. With probability 1 -γ , we have by Bernstein's inequality in Lemma D.1. Furthermore, at a fixed time h , we use Lemma C.3 to bound the gap between two consecutive appearances of the same missing pattern. We instantiate Lemma C.3 with θ = (1 -λ 0 ) h -t h -1 and obtain that the gap is bounded by ⌈ ι -log(1 -(1 -λ 0 ) h -t h -1 ) ⌉ with probability 1 -γ . Within the gap, the number of consecutive delays of length larger than h -t h -1 is bounded by

where inequality ( i ) follows from Bernstein's inequality again and inequality ( ii ) invokes the fact x +log(1 -x ) ≤ 0 for x ∈ [0 , 1) and bounds ⌈ x ⌉ by x +1 . Now we can bound the summation of the counting numbers. Conditioned on the event E m , we have

where inequality ( i ) follows since N k h is repeated at most C ≥ h -t h -1 times before getting an update and inequality ( ii ) follows from Cauchy-Schwarz inequality, and inequality ( iii ) invokes the assumption of λA ≤ 1 . Moreover, conditioned on the event E m , we also have

Combining the above On event E m , the regret is bounded by

where c is a sufficiently large constant and we substitute the bonus functions into inequality ( i ) . On the complement of E m , the regret is bounded by H (1 -P ( E m )) ≤ H 2 K (1 -λ 0 ) m +1 . We choose m = 1 2 ⌊ log K -log(1 -λ 0 ) ⌋ such that H (1 -P ( E m )) ≤ H 2 K (1 -λ 0 ) m +1 ≤ H 2 √ K . We can now check that A m +1 = exp ( log A -log(1 -λ 0 ) log √ K ) ≤ K 1 2(1+ v ) . Therefore, combining the regret on event E m and the complement event E ∁ m leads to

The proof is complete.

C.3 Supporting lemmas

Lemma C.2. Suppose Assumption 2.2 holds. With probability 1 -γ for some failure probability γ > 0 , we have

Proof of Lemma C.2. For any time h , we denote K eff ( h ) as the collection of episodes that the h -th and ( h +1) -th step observations are available. It is clear that the cardinality of K eff ( h ) is bounded by K for any h . Within each K eff ( h ) Thanks to Lemma C.3, the gap is bounded by q with probability 1 K (1 λ 2 ) q +1 . We set

K (1 -λ 2 0 ) q +1 = γ/H , which implies q = log HK γ -log(1 -λ 2 0 ) . Therefore, for any time step h , available observations are at most separated by q episodes.

, we would like to bound the gap between two observations. --0 ⌈ ⌉

With these notations, we bound

where inequality ( i ) follows since N k h will only be updated when h ∈ K eff ( h ) and then repeat at most ⌈ log HK γ -log(1 -λ 2 0 ) ⌉ times, inequality ( ii ) invokes the cardinality bound of K eff ( h ) , and inequality ( iii ) follows from the standard pigeon-hole principle.

Lemma C.3. Let { u i } k i =1 be i.i.d. Bernoulli random variables. Suppose P ( u i = 1) = θ . Define the largest gap between u i 's as

Then for any integer q > 0 , the following tail probability bound holds

Proof of Lemma C.3. We denote I neg = { ℓ 1 , . . . , ℓ m } as the index set for u ℓ i = 0 when i = 1 , . . . , | I neg | . Let v j = ℓ j +1 -ℓ j , which is a geometric random variable with a success rate θ . Note that the cardinality of I neg is at most k . Therefore, we have

where the last inequality follows from .

1 -kθ q +1 ≤ (1 -θ q +1 ) k

D Helper concentration inequalities

Lemma D.1 (Bernstein's inequality) . Let x 1 , . . . , x n be i.i.d. zero mean random variables. Suppose | x i | ≤ M for any i = 1 , . . . , n . Then for all positive t , it holds

In particular, given a failure probability γ < 1 , it holds

Proof of Lemma D.1. The proof of Bernstein's inequality is standard, see for example [Wainwright, 2019, Section 2.1]. Here we verify the second claim. Let exp ( -1 2 t 2 ∑ n i =1 Var[ x i ]+ 1 3 Mt ) ≤ γ hold true. We find a suitable t by

Lemma D.2 (Hoeffding's inequality) . Let x 1 , . . . , x n be i.i.d. random variables. Suppose a i ≤ x i ≤ b i for any i = 1 , . . . , n . Then for all positive t , it holds

In particular, given a failure probability γ < 1 , it holds

Proof of Lemma D.2. The proof is standard; see [Wainwright, 2019, Section 2.1].

Lemma D.3 (Azuma-Hoeffding's inequality) . Let x 1 , . . . , x n be a martingale adapted to filtration F 1 ⊂ · · · ⊂ F n . Suppose E [ x i -E [ x i ] |F i -1 ] = 0 and | x i -E [ x i ] | ≤ c i . Then for all positive t , it holds

In particular, given a failure probability γ < 1 , it holds

Proof of Lemma D.3. The proof is standard and applies Lemma D.2.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies online learning in tabular MDPs with impaired observability, which means one of two models: states are observed in stochastic delay or some observations are missing (sampled independently). The authors present algorithms based on optimism and prove regret bounds that scale optimally with $S,A,K$ and polynomially with $H$.\n\nSTRENGTHS:\n**Summary:** I don't have a strong conviction about this paper. On the one hand the paper is well-written and the model is interesting and novel, but on the other hand the regret bounds are straightforward and there is no interesting algorithmic novelty in my opinion.\n\n**Strengths:**\n1. RL with impaired observability is an interesting model which obviously has a lot of real-world applications. The authors do a good job of introducing the model, the challenges and the motivation.\n2. The paper is generally well-written and easy to follow (except for a few technical parts, see questions). It conveys well the need for the impaired observability model and the idea behind the algorithmic approaches to solve it.\n3. The algorithms are simple and the regret bounds are close to optimal (although one can argue about what is close to optimal in this setting).\n\nWEAKNESSES:\n**Weaknesses:**\n1. I think that the novelty of this paper is limited to the definition of the new models. In terms of algorithmic contribution, the application of optimism here is straightforward and so is the regret analysis. In fact, unless I am missing something, the augmented MDP is a factored MDP and then the algorithms (and analysis) actually exist already, so there is no surprise that the regret is polynomial. If that is the case, then the authors should also discuss the factored MDP literature, e.g., \"Near-optimal Reinforcement Learning in Factored MDPs\" by Ian Osband and Benjamin Van Roy.\n2. The algorithms run in exponential time in $H$. I understand that this is probably inevitable, but it still makes the contributions of this paper very limited.\n3. The regret analysis is similar to previous works and does not investigate in-depth the dependence on the delays or the missing observations rate. The regret of algorithm 2 \"hides\" the dependence in the distribution of the delays in the extra $H$ factors. I think that the actual dependence is interesting in this setting because it can show which delays actually become a problem. While the dependence on the missing rate is shown in the regret of algorithm 3, the optimality is not discussed and instead there is an additional algorithm that replaces this optimality with another assumption (that to me looks like it makes the whole thing easy because very few observations are missing). Moreover, There is not enough discussion about the assumptions 2.1 and 2.2, and what happens if they are relaxed.\n\nQUESTIONS:\n**Questions:**\n1. The definition of the augmented MDP is very hard to follow. Can that authors please explain in words what this MDP looks like?\n2. Can the authors please discuss the assumptions a little more? For example, is the assumption that feedback arrives in order necessary? what happens when there are large delays and why is the regret always bounded within $H$?\n3. Is the augmented MDP a factored MDP?\n4. Can you think of a scenario where the computational complexity is not exponential?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers reinforcement learning with delayed and missing observations. It is shown that a MDP with delayed observations is equivalent to an augmented MDP with perfect observations. Based on the augmented MDP, an optimistic algorithm is proposed for delayed MDPs and it is shown to achieve a near-optimal regret order. For MDPs with missing observations, two optimistic algorithms are proposed with near-optimal regret order under some assumptions on the missing rate.\n\nSTRENGTHS:\n- For MDPs with delayed observations, under some independent assumption and an assumption requiring the delayed observations will still arrive in order, the paper constructs an equivalent augmented MDP with perfect observations. Based on the augmented MDP, an optimistic algorithm is proposed for the delayed MDP. Despite the increased cardinality of the augmented MDP, the proposed algorithm is shown to achieve a regret with sharp dependence on the state and action spaces. \n\n- The effect of observation delay on performance degradation is discussed with some performance gap bounds provided between the optimal policy with and without observation delays. \n\n- For MDPs with missing observations, one optimistic algorithm is proposed and its regret is shown to be sub-linear in the number of episodes.\n\n- When the missing rate is small, another algorithm based on the augmented MDP is proposed. This algorithm is shown to have near-optimal regret order if the number of episode is large enough.\n\nWEAKNESSES:\n- There are some possible typos in key equations which may highly affect the derivations and results.\n  - In the definition of the transition below line 164, the collections of actions for time $h$ and $h+1$ may have different sizes. Is that just some typo or some additional things need to be handled?\n  - In the equation above line 182, should the reward be $r_{t_h}$?\n  - In the bonus function in Algorithm 2, $a_h$ seems to be a typo.\n \n- There seems to be several missing steps in the proof of Proposition 5.1. The entire derivation of the proof should be conditioned on some high probability event for the inequalities to hold, but those steps are missing. Inequality (i) is claimed to be true from valid optimism, but optimism for Algorithm 3 is not provided. Inequality (ii) might also need some further discussion.\n\nQUESTIONS:\n- Can the authors fix some of the typos and missing steps?\n\nLIMITATIONS:\nLimitations are adequately addressed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper aims to provide a theoretical analysis of reinforcement learning with delayed and missing state observations, and establish near-optimal regret bounds, for RL in both the delayed and missing observation settings. Despite impaired observability posing significant challenges to the policy class and planning, the results demonstrate that learning remains efficient, with the regret bound optimally depending on the state-action size of the original system.  The policy under impaired observability is evaluated.

STRENGTHS:
1. The problem addressed is critical for applying RL to real-world scenarios.
2. The theoretical analysis is comprehensive and well-executed.

WEAKNESSES:
1. The paper does not include preliminary experimental results, which may hinder the empirical understanding of the methods.
2. Building connections between the theoretical analysis and existing empirical studies in the literature would be beneficial.

QUESTIONS:
1. How can the theoretical analysis inspire further algorithm design to address impaired observations?
2. What are the main challenges preventing you from conducting empirical studies for the proposed method?

LIMITATIONS:
1. The scalability for large action spaces is limited.
2. The lack of empirical studies makes the applicability of the method uncertain.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies reinforcement learning in the setting where MDP is with delayed state information, but delayed observations still arrive in order, and in the setting where the state information of MDPs could be missed and never be observed. For both settings, this paper provides provably efficient RL algorithms with sub-linear regrets, whose dependencies on the number of episodes are optimal. In the setting with delayed state information, this paper shows the impact of delay $d_h$ on the performance of the optimal policy. In the setting with missing state information, this paper shows the impact of observation rate $\\lambda_h$ on the regret. The results seem interesting to me.\n\nSTRENGTHS:\n1. This paper studies MDPs with delayed state information and MDPs with missing state information, which seems to be an important problem.\n\n2. This paper provides algorithms for these two settings, and proves the regret guarantees.\n\n3. This paper shows the impacts of the values of delays and observation rates on the performance of the policies for learning in MDPs with impaired observability.\n\nWEAKNESSES:\n1. The novelty of the proposed algorithms is not clear to me. For example, algorithm 2 seems to be a simple application of the tabular RL algorithm to the case with delayed state information?\n\n2. It is not clear for me how to understand the regret. For example, why the benchmark in the regret is the optimal policy with impaired observability? Is it more reasonable to compare with stronger optimal policy, e.g., one with full observability? Since if the optimal policy is also with impaired observability, why is the problem more challenging?\n\nQUESTIONS:\n1. Why is the regret in Theorem 4.1 for delayed state information independent of delay? Is it because the optimal policy also suffers from the delay? Intuitively, the performance of the online policy with larger delay should be worse. For example, should we hope much better regret in the case with no delay than the case with infinite delay?\n\n2. Even though when the delay $d_h$ is 0, there is still a gap in the regret in Theorem 4.1? Could you give some insights whether this can actually be sharpened or why is this true?\n\n3. Similarly, in section 5 with missing state information, even when the missing rate is 0, the regret seems to keep to be larger than the best regret that could be obtained?\n\n4. Above line 119, the executable policy class is defined based on the action sequence. However, based on the notation, the action is always known, why the action sequence $a_{t_h:h-1}$ is important there?\n\nLIMITATIONS:\nPlease see weakness and questions above.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}