SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Minshuo Chen<sup>1</sup> Yu Bai<sup>2</sup> H. Vincent Poor<sup>1</sup> Mengdi Wang<sup>1</sup>  
<sup>1</sup>Princeton University Salesforce Research
In real-world reinforcement learning (RL) systems, various forms of impaired observability can complicate matters. These situations arise when an agent is unable to observe the most recent state of the system due to latency or lossy channels, yet the agent must still make real-time decisions. This paper introduces a theoretical investigation into efficient RL in control systems where agents must act with delayed and missing state observations. We establish near-optimal regret bounds, of the form  $\tilde{\mathcal{O}} (\sqrt{\mathrm{poly}(H)SAK})$ , for RL in both the delayed and missing observation settings. Despite impaired observability posing significant challenges to the policy class and planning, our results demonstrate that learning remains efficient, with the regret bound optimally depending on the state-action size of the original system. Additionally, we provide a characterization of the performance of the optimal policy under impaired observability, comparing it to the optimal value obtained with full observability.
In Reinforcement Learning (RL), an agent engages with an environment in a sequential manner. In an ideal setting, at each time step, the agent would observe the current state of the environment, select an action to perform, and receive a reward Smallwood and Sondik [1973], Bertsekas [2012], Sutton and Barto [2018], Lattimore and Szepesvári [2020]. However, real-world engineering systems often introduce impaired observability and latency, where the agent may not have immediate access to the instant state and reward information. In systems with lossy communication channels, certain state observations may even be permanently missing, never reaching the agent. Nevertheless, the agent is still required to make real-time decisions based on the available information.
The presence of impaired observability transforms the system into a complex interactive decision process (Figure 1), presenting challenges for both learning and planning in RL. With limited knowledge about recent states and rewards, the agent's policy must extract information from the observed history and utilize it to make immediate decisions. This introduces significant complexity to the policy class and poses difficulties for RL. Moreover, the loss of information due to permanently missing observations further hampers the efficiency of RL methods. Although a naïve approach would involve augmenting the state and action space to create a fully observable Markov Decision Process (MDP), such a method would lead to exponential regret growth in the state-action size.
Why existing methods do not work. One may be tempted to cast the problem of impaired observability into a Partially Observed MDPs (POMDPs). However, this would not solve the problem. In POMDP, the system does not reveal its instant state to the agent but provides an emission state observation conditioned on the latent state. POMDPs are known to suffer from the curse of history Papadimitriou and Tsitsiklis [1987], Bertsekas [2012], Krishnamurthy [2016], unless additional assumptions are imposed. Existing efficient algorithms focus on subclasses of POMDPs with decodable or distinguishable partial observations Jin et al. [2020], Uehara et al. [2022], Zhan et al.
Figure 1: Reinforcement learning with impaired observability. At time  $h$ , the agent only observes the past state  $s_{h - d}$  and actions  $a_{h - d}, \ldots, a_{h - 1}$ . The policy depends on the observed information.
[2022], Chen et al. [2022], Liu et al. [2022], Zhong et al. [2022], Chen et al. [2023], where the unseen instant state can be inferred from recent observations. Unfortunately, MDPs with impaired observability do not fall into these benign subclasses. The reason behind this is that at each time step, a new observation, if any, is in fact a past state. Viewing it as an emission state of the current one leads to a time reversal posterior distribution depending on the underlying transitions, which suffers from the curse of history and makes the POMDP intractable. The problem becomes even harder if some observations get missing.
Empirical evidences suggested that efficient RL is possible even with impaired state observability Lizotte et al. [2008], Liu et al. [2014], Agarwal and Aggarwal [2021]. However, theoretical understanding of this problem is very limited. One notable work Walsh et al. [2007] studied learning with constant-time delayed observations. They identified subclasses of MDPs with nearly deterministic transitions that can be efficiently learned. Beyond this special case, efficient RL with impaired observability in MDPs with fully generality remains largely open.
Some recent works studied delayed feedback in MDPs Yang et al. [2023], Howson et al. [2023]. It is a fundamentally different problem where the agent's policy can still access real-time states but learning uses delayed data. Our problem is fundamentally harder because the agent's policy can only access the lossy and delayed history. See Section 1 for more discussions.
Our results. In this paper, we provide algorithms and regret analysis for learning the optimal policy in tabular MDPs with impaired observability. Note that this optimal policy is a different one from the optimal policy with full observability. To approach this problem, we construct an augmented MDP reformulation where the original state space is expanded to include available observations of past state and an action sequence. However, the expanded state space is much larger than the original one and naive application of known methods would lead to exponentially large regret bounds. In our analysis, we exploit structure of the augmented transition model to achieve efficient learning and sharp regret bounds. The main results are summarized as follows.
- For MDPs with stochastic delays, we prove a sharp  $\widetilde{O}(H^4\sqrt{SAK})$  regret bound (Theorem 4.1) comparing to the best feasible policy, Here  $S$  and  $A$  are the sizes of the original state and action spaces, respectively,  $H$  is the horizon, and  $K$  is the number of episodes. Here we allows the delay to be stochastic and conditionally independent given on current state and action. Moreover, we quantify the performance degradation of optimal value due to impaired observability, compared to optimal value of fully observable MDPs (Proposition B.2). We also showcase in Proposition 4.2 that a short delay does not reduce the optimal value, but slightly longer delay leads to substantial degradation.
- For MDPs with randomly missing observations, we provide an optimistic RL method that provably achieves  $\tilde{\mathcal{O}} (\sqrt{H^3S^2AK})$  regret (Proposition 5.1). We also provide a sharper  $\tilde{\mathcal{O}} (H^4\sqrt{SAK})$  regret in the case when the missing rate is sufficiently small (Theorem 5.2).
To our best knowledge, these results present a first set of theories for RL with delayed and missing observations. Remarkably, our regret bounds nearly match the minimax-optimal regret of standard MDP in their dependence on  $S$ ,  $A$  (noting that the target optimal policies are different in the two cases). It implies that RL with impaired observability are provably as efficient as RL with full observability (up to poly factors of  $H$ ).
Related work. Efficient algorithms for learning in the standard setting of tabular MDPs without impaired observability has been extensively studied Kearns and Singh [2002], Brafman and Tennenholtz [2002], Jaksch et al. [2010], Dann and Brunskill [2015], Azar et al. [2017], Agrawal and Jia [2017],
Jin et al. [2018], Dann et al. [2019], Zanette and Brunskill [2019], Zhang et al. [2020], Domingues et al. [2021], where the minimax optimal regret is  $\tilde{\mathcal{O}} (\sqrt{H^3SAK})$  Azar et al. [2017], Domingues et al. [2021].
The delayed observation studied in this paper is related to delayed feedback in Howson et al. [2023], Yang et al. [2023], yet the setup is fundamentally different. In delayed feedback, an agent sends a policy to the environment for execution. The environment executes the policy on behalf of the agent for an episode, but the whole trajectory will be returned to the agent after some episodes. The policy executed by the environment is able to "see" instant state and reward. It is Markov and not played by the agent. Our setting concerns learning executable policies when delayed or missing states appear within an episode. The policy is no longer Markov and can only prescribe action based on history. Therefore, the algorithms and analyses for delayed feedback MDPs are not applicable to our settings.
Despite the distinct settings, there are existing fruitful results in efficiently learning MDPs or bandits with delayed feedback. Stochastic delayed feedback in bandits is studied in Agarwal and Duchi [2011], Dudik et al. [2011], Joulani et al. [2013], Vernade et al. [2017, 2020], Gael et al. [2020], Lancewicki et al. [2021]. In the more challenging setting of reinforcement learning, Howson et al. [2023] considers tabular MDPs and Yang et al. [2023] generalizes to MDPs with function approximation and multi-agent settings.
On the other hand, results analyzing MDPs with missing observations are limited in literature, although missing data is a commonly recognized issue in applications García-Laencina et al. [2010], Jerez et al. [2010], Little et al. [2012], Emmanuel et al. [2021]. One notable result is Bouneffouf et al. [2020] for bandits with missing rewards.
Notation: For real numbers  $a, b$ , we denote  $a \wedge b = \min \{a, b\}$ . In episodic MDPs, we use the superscript  $k$  to denote the index of episodes, and the subscript  $h$  to denote the index of time. We denote  $\mathbf{a}_{i:j} = \{a_i, \dots, a_j\}$  as the collection of actions from time  $i$  to  $j$ . For two probability distributions  $\mu$  and  $\nu$ , we denote their total variation distance as  $\| \mu - \nu \|_{\mathrm{TV}}$ .
MDP preliminary: An episodic MDP is described by a tuple  $(\mathcal{S},\mathcal{A},H,R,P)$ , where  $\mathcal{S},\mathcal{A}$  are state and action spaces, respectively,  $H$  is the horizon,  $R = \{r_h\}_{h = 1}^H$  is the reward function and  $P = \{p_h\}_{h = 1}^H$  is the transition probability. We primarily focus on tabular MDPs, where  $S = |\mathcal{S}|$  and  $A = |\mathcal{A}|$  are both finite. We also assume that the reward is uniformly bounded with  $\| r_h\|_{\infty}\leq 1$  for any  $h$ . An agent will interact with the environment for  $K$  episodes, hoping to find a good policy to maximize the cumulative reward. Within an episode, at the  $h$ -th step, the agent chooses an action based on the available information of the environment. After taking the action, the underlying environment produces a reward and transits to the next state. With full state observation, a policy  $\pi$  maps instant state  $s$  to an action  $a$  or an action distribution. Given such a policy  $\pi$ , the value function is  $V_{h}^{\pi}(s_{1}) = \mathbb{E}^{\pi}\left[\sum_{h^{\prime} = h}^{H}r_{h}(s_{h^{\prime}},a_{h^{\prime}})\big|s_{h}\right]$ , where  $\mathbb{E}^{\pi}$  is the policy induced expectation.
In this work, we study MDPs with impaired observability. We focus on two practical settings: 1) delayed observations and 2) missing observations.
2.1 MDP with delayed observations
In any episode, we denote  $d_h \in \{0, 1, \ldots\}$  as the observational delay of the state and reward at step  $h$ . That is, we receive  $s_h$  and  $r_h$  at time  $h + d_h$ . The delay time  $d_h$  can be dependent on the state  $s_h$  and action  $a_h$  at time  $h$ . To facilitate analysis, we denote the inter-arrival time between the arrival of observations for step  $h$  and  $h + 1$  as  $\Delta_h = d_{h+1} - d_h$ . With delays, at time  $h$ , the nearest observable state is denoted as  $s_{t_h}$ , where  $t_h = \operatorname{argmax}\left\{I : \sum_{i=0}^{I} \Delta_i \leq h\right\}$ . Then the executable policy class
$$
\Pi_ {\mathrm {e x e c}} = \left\{\pi_ {h} \left(\cdot \mid s _ {t _ {h}}, \mathbf {a} _ {t _ {h}; h - 1}\right) \text {f o r} h = 1, \dots , H \right\}
$$
chooses actions depending on the nearest visible state and history actions. We impose the following assumption on the interarrival time.
Assumption 2.1. The interarrival time  $\Delta_h$  takes value in  $\{0,1,\ldots\}$ . The distribution  $\mathcal{D}_h(s_h,a_h)$  of  $\Delta_h$  can depend on  $(s_h,a_h)$ , but is conditionally independent of the MDP transitions given  $(s_h,a_h)$ .
Assumption 2.1 does not impose any specific distributional assumption on  $\Delta_h$ , but only requires that the delayed observations arrive in order and at each time step, there is at most one new visible state and reward pair  $(\Delta_h \geq 0)$ . A widely studied example of delays in literature is that the inter-arrival time is geometrically distributed Winsten [1959]. Then the observation sequence  $\{h + d_h\}$  is known as a Bernoulli process, which is understood as the discretized version of a Poisson process.
Our delayed observation setting is newly proposed and substantially generalizes the Constant Delayed MDPs (CDMDPs) studied in Brooks and Leondes [1972], Bander and White III [1999], Katsikopoulos and Engelbrecht [2003], Walsh et al. [2007]. When  $\Delta_h = 0$  being deterministic for all  $h \geq 1$  and  $k$ , our observation delay coincides with CDMDPs. In CDMDPs, a new past observation is guaranteed to arrive at each time step. However, our delayed model can result in no new observation at some time steps.
Observation delay leads to difficulty in planning, as the agent can only infer the current state and then choose an action. Therefore, the policy is naturally history dependent. We summarize the interaction protocol of the agent with the environment in Protocol 1. At the end of each episode, we can collect
Protocol 1 Interaction between the agent and the environment with delayed observations
1: for episode  $k = 1, \ldots, K$  do  
2: for time  $h = 1, \ldots, H$  do  
3: The agent observes a pair of new, if any, state and reward  $(s_{t_h}^k, a_{t_h}^k)$ . By memory, the agent also has access to past actions  $\mathbf{a}_{t_h:h-1}^k$ .  
4: The agent plays action  $a_h^k$  according to some executable policy  $\pi_h^k \in \Pi_{\text{exec}}$ .  
5: The environment transits to next state  $s_{h+1}^k \sim p_h(\cdot | s_h^k, a_h^k)$ , which is unobservable to the agent. The environment also decides the delay at step  $h+1$  as  $d_{h+1}^k = d_h^k + \Delta_h^k$  and  $t_{h+1}^k$ .  
6: end for  
7: The environment sends all unobserved pairs of state and reward as well as their corresponding delay time to the agent.  
8: end for
all delayed observations, however, these observations are not used in planning. In reality, the agent can collect these observations by waiting after time  $H$ . Protocol 1 is similar to hindsight observability in POMDPs studied in Lee et al. [2023]. Yet their analysis for POMDPs is not directly transferable to our settings as mentioned in the introduction.
2.2 MDP with missing observations
In addition to the stochastic delay in observations, we also consider randomly missing observations. In applications, an agent interacts with the environment through some communication channel. The communication channel is often imperfect and thus, observation can be lost during transmission. This type of missing is permanent and we describe in the following assumption.
Assumption 2.2. Any pair of observation (state and reward) is independently observable in the communication channel. The observation rate is  $\lambda_{h}$  depending on  $h$ , but independent of the MDP transitions. Moreover, there exists a constant  $\lambda_0$  such that  $\lambda_h\geq \lambda_0$  for any  $h$ . The agent will be informed when an observation is missing.
Equivalently, the missing observation rate in Assumption 2.2 is  $1 - \lambda_h$  and assumes the upper bound of  $1 - \lambda_0$ . We will show later that this missing rate directly influences the learning efficiency in Section 5.
3 Construction of augmented MDPs
To tackle the limited observability, we expand the original state space and define an augmented MDP. It will serve as the basis for our subsequent theoretical analysis.
3.1 Augmented MDPs with expected reward
In the remainder of this section, we focus on the delayed observation case and defer the missing case to Section 5. Define  $\tau_{h} = \{s_{t_{h}},\mathbf{a}_{t_{h};h - 1},\delta_{t_{h}}\}$  as the augmented state, where  $\delta_{t_h}\in [0,\Delta_{t_h}]$  is the delayed steps after observing  $(s_{t_h},r_{t_h})$ . Let  $S_{\mathrm{aug}}$  denote the augmented state space of all possible  $\tau$ ’s. Then the original MDP with delayed observations can be reformulated into a state-augmented one  $\mathrm{MDP}_{\mathrm{aug}} = (\mathcal{S}_{\mathrm{aug}},\mathcal{A},H,R_{\mathrm{aug}},P_{\mathrm{aug}})$ . The reward is defined as
$$
r _ {h, \mathrm {a u g}} (\tau_ {h}, a _ {h}) = \mathbb {E} \left[ r _ {h} (s _ {h}, a _ {h}) | \tau_ {h}, a _ {h} \right],
$$
which is the expected reward given the nearest past state  $s_{t_h}$  and history actions  $\mathbf{a}_{t_h:h}$ . We can define belief distribution  $\mathfrak{b}_h(s|\tau_h) = \mathbb{P}(s_h = s|\tau_h)$ . Then  $r_{h,\mathrm{aug}}(\tau_h,a_h) = \mathbb{E}_{s\sim \mathfrak{b}_h(\cdot |\tau_h)}[r(s,a_h)]$ . Belief distributions are widely adopted in partially observed MDPs Ross et al. [2007], Poupart and Vlassis [2008]. We will frequently use the belief distribution to study the expressivity of  $\Pi_{\mathrm{exec}}$  in Section 4.2.
The transition probabilities  $P_{\mathrm{aug}}$  are sparse. For any  $\tau_h = \{s_{t_h}, \mathbf{a}_{t_h:h-1}, \delta_{t_h}\}$  and  $\tau_{h+1} = \{s_{t_{h+1}}, \mathbf{a}_{t_{h+1}:h}, \delta_{t_{h+1}}\}$ , we have
ph,aug(τh+1|τh,ah)	Condition
Ma(τh, τh+1)θdelay(sth, ath, δth)pth(sth+1|sth, ath)	if δth+1=0 and th+1=th+1
Ma(τh, τh+1)(1-θdelay(sth, ath, δth))	if δth+1=δth+1 and th+1=th
0	otherwise
where  $\mathsf{M}_a(\tau_h,\tau_{h + 1})$  indicates whether the rolling actions are matched, i.e.,
$$
\mathrm {M} _ {a} \left(\tau_ {h}, \tau_ {h + 1}\right) = \mathbb {1} \left\{\mathbf {a} _ {t _ {h}: h - 1} = \mathbf {a} _ {t _ {h + 1}: h - 1} \right\},
$$
and  $\theta_{\mathrm{delay}}(s_{t_h},a_{t_h},\delta_{t_h})$  is defined as
$$
\theta_ {\mathrm {d e l a y}} (s _ {t _ {h}}, a _ {t _ {h}}, \delta_ {t _ {h}}) = \mathbb {P} (\Delta_ {t _ {h}} = \delta_ {t _ {h}} | s _ {t _ {h}}, a _ {t _ {h}}, \delta_ {t _ {h}}) = \frac {\mathbb {P} (\Delta_ {t _ {h}} = \delta_ {t _ {h}} | s _ {t _ {h}} , a _ {t _ {h}})}{1 - \sum_ {\delta <   \delta_ {t _ {h}}} \mathbb {P} (\Delta_ {t _ {h}} = \delta | s _ {t _ {h}} , a _ {t _ {h}})}.
$$
The factored form of  $\theta_{\mathrm{delay}}(s_{t_h},a_{t_h},\delta_{t_h})p_{t_h}(s_{t_{h + 1}}|s_{t_h},a_{t_h})$  follows from the conditional independence in Assumption 2.1. We define  $Q$ -functions and value functions as follows. For any  $\tau_h,a_h$  and policy  $\pi \in \Pi_{\mathrm{exec}}$ , we have
$$
Q _ {h, \text {a u g}} ^ {\pi} \left(\tau_ {h}, a _ {h}\right) = \mathbb {E} ^ {\pi} \left[ \sum_ {h ^ {\prime} = h} ^ {H} r _ {h, \text {a u g}} \left(\tau_ {h ^ {\prime}}, a _ {h ^ {\prime}}\right) \mid \tau_ {h}, a _ {h} \right] \quad \text {a n d}
$$
$$
V _ {h, \mathrm {a u g}} ^ {\pi} (\tau_ {h}) = \left\langle Q _ {h, \mathrm {a u g}} ^ {\pi} (\tau_ {h}, \cdot), \pi_ {h} (\cdot | \tau_ {h}) \right\rangle .
$$
We note that  $V_{h}^{\pi}$  is equivalent to  $V_{h,\mathrm{aug}}^{\pi}$  for the same executable policy  $\pi \in \Pi_{\mathrm{exec}}$ . We also denote  $\mathcal{P}_{h,\mathrm{aug}}$  as the transition operator corresponding to  $P_{\mathrm{aug}}$ . It can be checked that
$$
Q _ {h, \mathrm {a u g}} ^ {\pi} \left(\tau_ {h}, a _ {h}\right) = r _ {h, \mathrm {a u g}} \left(\tau_ {h}, a _ {h}\right) + \left[ \mathcal {P} _ {h, \mathrm {a u g}} V _ {h, \mathrm {a u g}} ^ {\pi} \right] \left(\tau_ {h}, a _ {h}\right).
$$
$\mathrm{MDP}_{\mathrm{aug}}$  also appears in makes all the policies in  $\Pi_{\mathrm{exec}}$  executable and Markov. Meanwhile, the reward function keeps track of all the expected reward for  $H$  steps. Although the expanded state space  $S_{\mathrm{aug}}$  is much more complicated than the original state space  $S$ , the sparse structures in the transition probabilities still allow an efficient exploration. We note that  $p_{h,\mathrm{aug}}$  only depends on the delay distribution and one-step Markov transitions. However, there is still one caveat for learning in  $\mathrm{MDP}_{\mathrm{aug}}$  - the reward function depends belief distributions, which involve multi-step transitions.
3.2 Augmented MDPs with past reward
To tackle the aforementioned challenge, we further define  $\widetilde{\mathrm{MDP}}_{\mathrm{aug}} = (\widetilde{S}_{\mathrm{aug}},\mathcal{A},\widetilde{H},\widetilde{R}_{\mathrm{aug}},\widetilde{P}_{\mathrm{aug}})$  that shares the optimal policy in  $\mathrm{MDP}_{\mathrm{aug}}$  with an enlarged horizon  $\widetilde{H} = 2H$ . The state space  $\widetilde{S}_{\mathrm{aug}}$  consists of any  $\tau_h = \{s_{t_h},\mathbf{a}_{t_h:h\wedge H},\delta_{t_h}\}$ . Comparing to  $S_{\mathrm{aug}}$ , we cut off the action at horizon  $H$ , since  $a_{h}$  for  $h > H$  has no influence on the state and reward in time  $[1,H]$ . The reward function is defined as
$$
\widetilde {r} _ {h, \mathrm {a u g}} \left(\tau_ {h}, a _ {h}\right) = r _ {t _ {h}} \left(s _ {t _ {h}}, a _ {t _ {h}}\right) \mathbb {1} \left\{\delta_ {t _ {h}} = 0 \right\} \mathbb {1} \left\{t _ {h} \in \{1, \dots , H \} \right\}.
$$
By definition,  $\widetilde{r}_{\mathrm{aug}}(\tau_h,a_h)$  is a past reward. More importantly,  $\widetilde{r}_{h,\mathrm{aug}}(\tau_h,a_h)$  zeros out rewards outside the original horizon  $H$ . Meanwhile, between the arrival of two consecutive state observations, the reward only counts once. Lastly, the transition probabilities are
\(\widetilde{p}_{h,\text{aug}}(\tau_{h+1}|\tau_{h},a_{h})\)	Condition
\(M_{a}(\tau_{h},\tau_{h+1})\theta_{\text{delay}}(s_{th},a_{th},\delta_{th})p_{t_{h}}(s_{th+1}|s_{th},a_{th})\)	if \(\delta_{t_{h+1}}=0,t_{h+1}=t_{h}+1\) and \(h< H\)
\(M_{a}(\tau_{h},\tau_{h+1})(1-\theta_{\text{delay}}(s_{th},a_{th},\delta_{th}))\)	if \(\delta_{t_{h+1}}=\delta_{t_{h}}+1,t_{h+1}=t_{h}\) and \(h< H\)
\(M_{a}(\tau_{h},\tau_{h+1})p_{t_{h}}(s_{th+1}|s_{th},a_{th})\)	if \(\delta_{t_{h+1}}=0,t_{h+1}=t_{h}+1\) and \(h>H\)
0	otherwise
We interpret the transitions as follows. When  $h \leq H$ , the transition is the same as  $\mathrm{MDP}_{\mathrm{aug}}$ . When  $h > H$ , we simply wait for unobserved states and rewards to come. As mentioned, actions taken beyond time  $H$  are irrelevant. We build an equivalence in the expected values of  $\mathrm{MDP}_{\mathrm{aug}}$  and  $\widetilde{\mathrm{MDP}}_{\mathrm{aug}}$ .
Proposition 3.1. Let  $\mathsf{MDP}_{\mathrm{aug}}$  and  $\widetilde{\mathsf{MDP}}_{\mathrm{aug}}$  be defined as in the previous paragraphs. Then for any initial state  $\tau_{1}$  and any policy  $\pi = \{\pi_h\}_{h=1}^H \in \Pi_{\mathrm{exec}}$ , it holds that
$$
\mathbb {E} ^ {\pi} \left[ \sum_ {h = 1} ^ {H} r _ {h, \text {a u g}} \left(\tau_ {h}, a _ {h}\right) \mid \tau_ {1} \right] = \mathbb {E} ^ {\pi} \left[ \sum_ {h = 1} ^ {\widetilde {H}} \widetilde {r} _ {h, \text {a u g}} \left(\tau_ {h}, a _ {h}\right) \mid \tau_ {1} \right],
$$
where in the right-hand side, the policy for steps  $H + 1$  to  $\widetilde{H}$  is arbitrary.
The proof is provided in Appendix A.1. Proposition 3.1 implies that learning in  $\mathrm{MDP}_{\mathrm{aug}}$  until time  $H$  is equivalent to that in  $\widetilde{\mathrm{MDP}}_{\mathrm{aug}}$  for  $\widetilde{H}$  steps.
4 RL with delayed observations and regret bound