SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \left[ \prod_ {k = 1} ^ {t} \mathbb {P} \left(z _ {k}, x _ {k} \mid x _ {k - 1}, \pi_ {k - 1}\right) b \left(x _ {0}\right) - \prod_ {k = 1} ^ {t - 1} b \left(x _ {0}\right) \overline {{\mathbb {P}}} \left(z _ {k}, x _ {k} \mid x _ {k - 1}, \pi_ {k - 1}\right) \mathbb {P} \left(z _ {t}, x _ {t} \mid x _ {t - 1}, \pi_ {t - 1}\right) \right. \\ + \prod_ {k = 1} ^ {t - 1} b (x _ {0}) \overline {{\mathbb {P}}} \left(z _ {k}, x _ {k} \mid x _ {k - 1}, \pi_ {k - 1}\right) \mathbb {P} \left(z _ {t}, x _ {t} \mid x _ {t - 1}, \pi_ {t - 1}\right) - \prod_ {k ^ {\prime} = 1} ^ {t} b (x _ {0}) \overline {{\mathbb {P}}} \left(z _ {k ^ {\prime}}, x _ {k ^ {\prime}} \mid x _ {k ^ {\prime} - 1}, \pi_ {k ^ {\prime} - 1}\right) ] \\ = \sum_ {z _ {1: t}} \sum_ {x _ {0: t}} | r \left(x _ {t}\right) | \Big \} \tag {31} \\ \mathbb {P} \left(z _ {t}, x _ {t} \mid x _ {t - 1}, \pi_ {t - 1}\right) \left[ \prod_ {k = 1} ^ {t - 1} \mathbb {P} \left(z _ {k}, x _ {k} \mid x _ {k - 1}, \pi_ {k - 1}\right) b \left(x _ {0}\right) - \prod_ {k = 1} ^ {t - 1} b \left(x _ {0}\right) \overline {{\mathbb {P}}} \left(z _ {k}, x _ {k} \mid x _ {k - 1}, \pi_ {k - 1}\right) \right] \\ \left. + \prod_ {k = 1} ^ {t - 1} b (x _ {0}) \overline {{\mathbb {P}}} \left(z _ {k}, x _ {k} \mid x _ {k - 1}, \pi_ {k - 1}\right) \left[ \mathbb {P} \left(z _ {t}, x _ {t} \mid x _ {t - 1}, \pi_ {t - 1}\right) - \overline {{\mathbb {P}}} \left(z _ {t}, x _ {t} \mid x _ {t - 1}, \pi_ {t - 1}\right) \right] \right\} \\ \end{array}
$$
applying Holder's inequality,
$$
\begin{array}{l} \leq \mathcal {R} _ {\max } \sum_ {z _ {1: t}} \sum_ {x _ {0: t}} \mathbb {P} \left(z _ {t}, x _ {t} \mid x _ {t - 1}, \pi_ {t - 1}\right) \left[ b \left(x _ {0}\right) \prod_ {k = 1} ^ {t - 1} \mathbb {P} \left(z _ {k}, x _ {k} \mid x _ {k - 1}, \pi_ {k - 1}\right) - b \left(x _ {0}\right) \prod_ {k = 1} ^ {t - 1} \overline {{\mathbb {P}}} \left(z _ {k}, x _ {k} \mid x _ {k - 1}, \pi_ {k - 1}\right) \right] (32) \\ + \mathcal {R} _ {\max } \sum_ {z _ {1: t}} \sum_ {x _ {0: t}} \prod_ {k = 1} ^ {t - 1} \overline {{\mathbb {P}}} (z _ {k}, x _ {k} \mid x _ {k - 1}, \pi_ {k - 1}) b (x _ {0}) [ \mathbb {P} (z _ {t}, x _ {t} \mid x _ {t - 1}, \pi_ {t - 1}) - \overline {{\mathbb {P}}} (z _ {t}, x _ {t} \mid x _ {t - 1}, \pi_ {t - 1}) ] \\ = \mathcal {R} _ {\max } \sum_ {z _ {1: t}} \sum_ {x _ {0: t}} \mathbb {P} \left(z _ {t}, x _ {t} \mid x _ {t - 1}, \pi_ {t - 1}\right). (33) \\ \left[ b (x _ {0}) \prod_ {k = 1} ^ {t - 1} \mathbb {P} \left(z _ {k}, x _ {k} \mid x _ {k - 1}, \pi_ {k - 1}\right) - b (x _ {0}) \prod_ {k = 1} ^ {t - 1} \overline {{\mathbb {P}}} \left(z _ {k}, x _ {k} \mid x _ {k - 1}, \pi_ {k - 1}\right) \right] + \mathcal {R} _ {\max } \delta_ {t} \\ = \mathcal {R} _ {\max } \sum_ {z _ {1: t - 1}} \sum_ {x _ {0: t - 1}} \left[ b \left(x _ {0}\right) \prod_ {k = 1} ^ {t - 1} \mathbb {P} \left(z _ {k}, x _ {k} \mid x _ {k - 1}, \pi_ {k - 1}\right) - b \left(x _ {0}\right) \prod_ {k = 1} ^ {t - 1} \overline {{\mathbb {P}}} \left(z _ {k}, x _ {k} \mid x _ {k - 1}, \pi_ {k - 1}\right) \right] (34) \\ + \mathcal {R} _ {\max } \delta_ {t}, \\ \end{array}
$$
following similar steps recursively,
$$
= \dots = \mathcal {R} _ {\max } \sum_ {\tau = 1} ^ {t} \delta_ {\tau}. \tag {35}
$$
Finally, applying similar steps for every time step  $t \in [1, T]$  results in,
$$
\left| V ^ {\pi} \left(b _ {t}\right) - \bar {V} ^ {\pi} \left(b _ {t}\right) \right| \leq \mathcal {R} _ {\max } \sum_ {t = 1} ^ {T} \sum_ {\tau = 1} ^ {t} \delta_ {\tau} \tag {36}
$$
$$
\begin{array}{l} \delta_ {\tau} = \sum_ {z _ {1: \tau}} \sum_ {x _ {0: \tau}} \prod_ {k = 1} ^ {\tau - 1} \overline {{\mathbb {P}}} (z _ {k}, x _ {k} \mid x _ {k - 1}, \pi_ {k - 1}) b (x _ {0}) [ \mathbb {P} (z _ {\tau}, x _ {\tau} \mid x _ {\tau - 1}, \pi_ {\tau - 1}) - \overline {{\mathbb {P}}} (z _ {\tau}, x _ {\tau} \mid x _ {\tau - 1}, \pi_ {\tau - 1}) ] \\ = \sum_ {z _ {1: \tau - 1}} \sum_ {x _ {0: \tau - 1}} \prod_ {k = 1} ^ {\tau - 1} \overline {{\mathbb {P}}} \left(z _ {k}, x _ {k} \mid x _ {k - 1}, \pi_ {k - 1}\right) b \left(x _ {0}\right) [ 1 - \sum_ {z _ {\tau}} \sum_ {x _ {\tau}} \overline {{\mathbb {P}}} \left(z _ {\tau}, x _ {\tau} \mid x _ {\tau - 1}, \pi_ {\tau - 1}\right) ] \tag {37} \\ \end{array}
$$
plugging the term in (37) to (36) and expanding the terms results in the desired bound,
$$
\left| V ^ {\pi} \left(b _ {t}\right) - \bar {V} ^ {\pi} \left(b _ {t}\right) \right| \leq \mathcal {R} _ {\max } \sum_ {\tau = t + 1} ^ {T} \left[ 1 - \sum_ {z _ {t + 1: \tau}} \sum_ {x _ {t: \tau}} b \left(x _ {t}\right) \prod_ {k = t + 1} ^ {\tau} \overline {{\mathbb {P}}} \left(z _ {k} \mid x _ {k}\right) \mathbb {P} \left(x _ {k} \mid x _ {k - 1}, \pi_ {k - 1}\right) \right] \tag {38}
$$
which concludes our derivation.
7.2 Lemma 1
Lemma 2. The optimal value function can be bounded as
$$
V ^ {\pi *} \left(b _ {t}\right) \leq \mathrm {U D B} ^ {\pi} \left(b _ {t}\right), \tag {39}
$$
$$
\begin{array}{l} \mathrm {U D B} ^ {\pi} \left(b _ {t}\right) \triangleq \max  _ {a _ {t} \in \mathcal {A}} \left[ \bar {Q} ^ {\pi} \left(b _ {t}, a _ {t}\right) + \epsilon_ {z} ^ {\pi} \left(b _ {t}, a _ {t}\right) \right] (40) \\ = \max  _ {a _ {t} \in \mathcal {A}} \left[ r \left(b _ {t}, a _ {t}\right) + \bar {\mathbb {E}} _ {z _ {t + 1} \mid b _ {t}, a _ {t}} \left[ \bar {V} ^ {\pi} \left(b _ {t + 1}\right) \right] + \epsilon_ {z} ^ {\pi} \left(b _ {t}, a _ {t}\right) \right]. (41) \\ \end{array}
$$
Proof. In the following, we prove by induction that applying the Bellman optimality operator on upper bounds to the value function in finite-horizon POMDPs will result in an upper bound on the optimal value function. The notations are the same as the ones presented in the main body of the paper. We restate some of the definitions from the paper for convenience.
The policy  $\pi_t(b_t)$  determined by applying Bellman optimality at belief  $b_{t}$ , i.e.,
$$
\pi_ {t} \left(b _ {t}\right) = \arg \max  _ {a _ {t} \in \mathcal {A}} \left[ \bar {Q} ^ {\pi} \left(b _ {t}, a _ {t}\right) + \epsilon_ {z} ^ {\pi} \left(b _ {t}, a _ {t}\right) \right]. \tag {42}
$$
As it will be needed in the following proof, we also define the value of a belief which includes in its history at least one observation out of the simplified set, e.g.  $H_{t} = \{a_{0},z_{1},\ldots ,z_{k}\notin \overline{\mathcal{Z}},\ldots ,z_{t}\}$  as being equal to zero. Explicitly,
$$
\bar {V} _ {t} ^ {\pi} \left(\mathbb {P} \left(x _ {t} \mid a _ {0}, z _ {1}, \dots , z _ {k} \notin \bar {\mathcal {Z}}, \dots , z _ {t}\right)\right) \equiv 0 \forall k \in [ 1, t ]. \tag {43}
$$
We also use the following simple bound,
$$
V _ {t, \max } \triangleq \mathcal {R} _ {\max } \cdot (T - t - 1) \tag {44}
$$
Base case ( $t = T$ ) - At the final time step  $T$ , for each belief we set the value function to be equal to the reward value at that belief state,  $b_{T}$  and taking the action that maximizes the immediate reward,
$$
\mathrm {U D B} ^ {\pi} \left(b _ {T}\right) = \max  _ {a _ {T}} \left\{r \left(b _ {T}, a _ {T}\right) + \epsilon_ {z} \left(b _ {T}, a _ {T}\right) \right\} \equiv \arg \max  _ {a _ {T}} \left\{r \left(b _ {T}, a _ {T}\right) \right\} \tag {45}
$$
which provides an upper bound for the optimal value function for the final time step,  $V_T^\star(b_T) \leq \mathrm{UDB}^\pi(b_T)$ .
Induction hypothesis - Assume that for a given time step,  $t$ , for all belief states the following holds,
$$
V _ {t} ^ {\star} \left(b _ {t}\right) \leq \mathrm {U D B} ^ {\pi} \left(b _ {t}\right). \tag {46}
$$
Induction step - We will show that the hypothesis holds for time step  $t - 1$ . By the induction hypothesis,
$$
V _ {t} ^ {\star} \left(b _ {t}\right) \leq \mathrm {U D B} ^ {\pi} \left(b _ {t}\right) \forall b _ {t}, \tag {47}
$$
$$
\begin{array}{l} Q ^ {\star} \left(b _ {t - 1}, a _ {t - 1}\right) = r \left(b _ {t - 1}, a _ {t - 1}\right) + \sum_ {z _ {t} \in \mathcal {Z}} \mathbb {P} \left(z _ {t} \mid H _ {t} ^ {-}\right) V _ {t} ^ {\star} \left(b \left(z _ {t}\right)\right) (48) \\ \leq r \left(b _ {t - 1}, a _ {t - 1}\right) + \sum_ {z _ {t} \in \mathcal {Z}} \mathbb {P} \left(z _ {t} \mid H _ {t} ^ {-}\right) \mathrm {U D B} ^ {\pi} \left(b \left(z _ {t}\right)\right) (49) \\ = r \left(b _ {t - 1}, a _ {t - 1}\right) + \sum_ {z _ {t} \in \mathcal {Z}} \mathbb {P} \left(z _ {t} \mid H _ {t} ^ {-}\right) \left[ \overline {{V}} _ {t} ^ {\pi} \left(b _ {t}\right) + \epsilon_ {z} ^ {\pi} \left(b _ {t}\right) \right]. (50) \\ \end{array}
$$
For the following transition, we make use of lemma 3,
$$
\begin{array}{l} = r \left(b _ {t - 1}, a _ {t - 1}\right) + \overline {{\mathbb {E}}} _ {z _ {t} \mid b _ {t - 1}, a _ {t - 1}} \left[ \bar {V} _ {t} ^ {\pi} \left(b _ {t}\right) \right] + \epsilon_ {z} ^ {\pi} \left(b _ {t - 1}, a _ {t - 1}\right) (51) \\ \equiv \mathrm {U D B} ^ {\pi} \left(b _ {t - 1}, a _ {t - 1}\right). (52) \\ \end{array}
$$
Therefore, under the induction hypothesis,  $Q_{t-1}^{\star}(b_{t-1}, a_{t-1}) \leq \mathrm{UDB}^{\pi}(b_{t-1}, a_{t-1})$ . Taking the maximum over all actions  $a_t$ ,
$$
\begin{array}{l} \mathrm {U D B} ^ {\pi} \left(b _ {t - 1}\right) = \max  _ {a _ {t - 1} \in \mathcal {A}} \left\{\mathrm {U D B} ^ {\pi} \left(b _ {t - 1}, a _ {t - 1}\right) \right\} \tag {53} \\ \geq \max _ {a _ {t - 1} \in \mathcal {A}} \left\{Q _ {t - 1} ^ {\star} (b _ {t - 1}, a _ {t - 1}) \right\} = V _ {t - 1} ^ {\star} (b _ {t - 1}), \\ \end{array}
$$
which completes the induction step and the required proof.
Lemma 3. Let  $b_{t}$  denote a belief state and  $\pi_t$  a policy at time  $t$ . Let  $\bar{\mathbb{P}}(z_t \mid x_t)$  be the simplified observation model which represents the likelihood of observing  $z_t$  given  $x_t$ . Then, the following terms are equivalent,
$$
\mathbb {E} _ {z _ {t}} \left[ \bar {V} _ {t} ^ {\pi} (b _ {t}) + \epsilon_ {z} ^ {\pi} (b _ {t}) \right] = \bar {\mathbb {E}} _ {z _ {t}} \left[ \bar {V} _ {t} ^ {\pi} (b _ {t}) \right] + \epsilon_ {z} ^ {\pi} (b _ {t - 1}, a _ {t - 1}) \tag {54}
$$
$$
\mathbb {E} _ {z _ {t}} \left[ \bar {V} _ {t} ^ {\pi} (b _ {t}) + \epsilon_ {z} ^ {\pi} (b _ {t}) \right] = \tag {55}
$$
$$
\mathbb {E} _ {z _ {t}} \left[ \overline {{V}} _ {t} ^ {\pi} \left(b _ {t}\right) \right] + \mathbb {E} _ {z _ {t}} \left[ \mathcal {R} _ {\max } \sum_ {\tau = t + 1} ^ {T} \left[ 1 - \sum_ {z _ {t + 1: \tau}} \sum_ {x _ {t: \tau}} b _ {t} \prod_ {k = t + 1} ^ {\tau} \overline {{\mathbb {P}}} \left(z _ {k} \mid x _ {k}\right) \mathbb {P} \left(x _ {k} \mid x _ {k - 1}, \pi_ {k - 1}\right) \right] \right] \tag {56}
$$
focusing on the second summand,
$$
\begin{array}{l} \sum_ {z _ {t} \in \mathcal {Z}} \mathbb {P} \left(z _ {t} \mid H _ {t} ^ {-}\right) \mathcal {R} _ {\max } \sum_ {\tau = t + 1} ^ {T} \left[ 1 - \sum_ {z _ {t + 1: \tau}} \sum_ {x _ {t: \tau}} b _ {t} \prod_ {k = t + 1} ^ {\tau} \overline {{\mathbb {P}}} \left(z _ {k} \mid x _ {k}\right) \mathbb {P} \left(x _ {k} \mid x _ {k - 1}, \pi_ {k - 1}\right) \right] (57) \\ = \mathcal {R} _ {\max } \sum_ {\tau = t + 1} ^ {T} \left[ 1 - \sum_ {z _ {t}} \mathbb {P} \left(z _ {t} \mid H _ {t} ^ {-}\right) \sum_ {z _ {t + 1: \tau}} \sum_ {x _ {t: \tau}} b \left(x _ {t}\right) \prod_ {k = t + 1} ^ {\tau} \overline {{\mathbb {P}}} \left(z _ {k} \mid x _ {k}\right) \mathbb {P} \left(x _ {k} \mid x _ {k - 1}, \pi_ {k - 1}\right) \right] (58) \\ \end{array}
$$
by marginalizing over  $x_{t - 1}$
$$
\begin{array}{l} = \mathcal {R} _ {\max } \sum_ {\tau = t + 1} ^ {T} \left[ 1 - \sum_ {z _ {t}} \mathbb {P} \left(z _ {t} \mid H _ {t} ^ {-}\right) \sum_ {z _ {t + 1: \tau}} \sum_ {x _ {t - 1: \tau}} \frac {\overline {{\mathbb {P}}} \left(z _ {t} \mid x _ {t}\right) \mathbb {P} \left(x _ {t} \mid x _ {t - 1} , \pi_ {t - 1}\right) b \left(x _ {t - 1}\right)}{\mathbb {P} \left(z _ {t} \mid H _ {t} ^ {-}\right)}. \right. \tag {59} \\ \prod_ {k = t + 1} ^ {\tau} \overline {{\mathbb {P}}} (z _ {k} \mid x _ {k}) \mathbb {P} (x _ {k} \mid x _ {k - 1}, \pi_ {k - 1}) ] \\ \end{array}
$$
canceling out the denominator,
$$
\begin{array}{l} = \mathcal {R} _ {\max } \sum_ {\tau = t + 1} ^ {T} [ 1 - \sum_ {z _ {t: \tau}} \sum_ {x _ {t - 1: \tau}} \overline {{\mathbb {P}}} (z _ {t} \mid x _ {t}) \mathbb {P} (x _ {t} \mid x _ {t - 1}, a _ {t - 1}) b (x _ {t - 1}). \tag {60} \\ \prod_ {k = t + 1} ^ {\tau} \overline {{\mathbb {P}}} (z _ {k} \mid x _ {k}) \mathbb {P} (x _ {k} \mid x _ {k - 1}, \pi_ {k - 1}) ] \equiv \epsilon_ {z} ^ {\pi} (b _ {t - 1}, a _ {t - 1}) \\ \end{array}
$$
it is left to show that  $\mathbb{E}_{z_t|b_{t - 1},a_{t - 1}}\left[\overline{V}_t^\pi (b_t)\right] = \overline{\mathbb{E}}_{z_t|b_{t - 1},a_{t - 1}}\left[\overline{V}_t^\pi (b_t)\right]$ . By the definition of a value function of a belief not included in the simplified set, we have that,
$$
\begin{array}{l} \mathbb {E} _ {z _ {t} \mid b _ {t - 1}, a _ {t - 1}} \left[ \bar {V} _ {t} ^ {\pi} \left(b _ {t}\right) \right] = \sum_ {z _ {t} \in \mathcal {Z}} \mathbb {P} \left(z _ {t} \mid H _ {t} ^ {-}\right) \bar {V} _ {t} ^ {\pi} \left(b _ {t}\right) (61) \\ = \sum_ {z _ {t} \in \overline {{\mathcal {Z}}}} \mathbb {P} \left(z _ {t} \mid H _ {t} ^ {-}\right) \bar {V} _ {t} ^ {\pi} (b _ {t}) + \sum_ {z _ {t} \in \mathcal {Z} \backslash \overline {{\mathcal {Z}}}} \mathbb {P} \left(z _ {t} \mid H _ {t} ^ {-}\right) \bar {V} _ {t} ^ {\pi} (b _ {t}) (62) \\ = \sum_ {z _ {t} \in \overline {{\mathcal {Z}}}} \bar {\mathbb {P}} \left(z _ {t} \mid H _ {t} ^ {-}\right) \cdot \bar {V} _ {t} ^ {\pi} (b _ {t}) + \sum_ {z _ {t} \in \mathcal {Z} \backslash \overline {{\mathcal {Z}}}} \mathbb {P} \left(z _ {t} \mid H _ {t} ^ {-}\right) \cdot 0 (63) \\ = \overline {{\mathbb {E}}} _ {z _ {t} \mid b _ {t - 1}, a _ {t - 1}} \left[ \bar {V} _ {t} ^ {\pi} (b _ {t}) \right], (64) \\ \end{array}
$$
which concludes the derivation.
7.3 Corollary 1.1
We restate the definition of UDB exploration criteria,
$$
a _ {t} = \arg \max  _ {a _ {t} \in \mathcal {A}} \left[ \mathrm {U D B} ^ {\pi} \left(b _ {t}, a _ {t}\right) \right] = \arg \max  _ {a _ {t} \in \mathcal {A}} \left[ \bar {Q} ^ {\pi} \left(b _ {t}, a _ {t}\right) + \epsilon_ {z} ^ {\pi} \left(b _ {t}, a _ {t}\right) \right]. \tag {65}
$$
Corollary 3.1. Using Lemma 2 and the exploration criteria defined in (65) guarantees convergence to the optimal value function.
Proof. Let us define a sequence of bounds,  $\mathrm{UDB}_n^\pi (b_t)$  and a corresponding difference value between  $\mathrm{UDB}_n$  and the simplified value function,
$$
\mathrm {U D B} _ {n} ^ {\pi} \left(b _ {t}\right) - \bar {V} _ {n} ^ {\pi} \left(b _ {t}\right) = \epsilon_ {n, z} ^ {\pi} \left(b _ {t}\right), \tag {66}
$$
where  $n \in [0,|\mathcal{Z}|]$  corresponds to the number of unique observation instances within the simplified observation set,  $\overline{\mathcal{Z}}_n$ , and  $|\mathcal{Z}|$  denotes the cardinality of the complete observation space. Additionally, for the clarity of the proof and notations, assume that by construction the simplified set is chosen such that  $\overline{\mathcal{Z}}_n(H_t) \equiv \overline{\mathcal{Z}}_n$  remains identical for all time steps  $t$  and history sequences,  $H_{t}$  given  $n$ . By the definition of  $\epsilon_{n,z}^{\pi}(b_t)$ ,
$$
\epsilon_ {n, z} ^ {\pi} \left(b _ {t}\right) = \mathcal {R} _ {\max } \sum_ {\tau = t + 1} ^ {T} \left[ 1 - \sum_ {z _ {t + 1: \tau} \in \overline {{\mathcal {Z}}} _ {n}} \sum_ {x _ {t: \tau}} b \left(x _ {t}\right) \prod_ {k = t + 1} ^ {\tau} \overline {{\mathbb {P}}} \left(z _ {k} \mid x _ {k}\right) \mathbb {P} \left(x _ {k} \mid x _ {k - 1}, \pi_ {k - 1}\right) \right], \tag {67}
$$
we have that  $\epsilon_{n,z}^{\pi}(b_t) \to 0$  as  $n \to |\mathcal{Z}|$ , since
$$
\sum_ {z _ {t + 1: \tau} \in \overline {{\mathcal {Z}}} _ {n}} \sum_ {x _ {t: \tau}} b \left(x _ {t}\right) \prod_ {k = t + 1} ^ {\tau} \bar {\mathbb {P}} \left(z _ {k} \mid x _ {k}\right) \mathbb {P} \left(x _ {k} \mid x _ {k - 1}, \pi_ {k - 1}\right)\rightarrow 1 \tag {68}
$$
as more unique observation elements are added to the simplified observation space,  $\overline{\mathcal{Z}}_n$ , eventually recovering the entire support of the discrete observation distribution.
From lemma 2 we have that, for all  $n \in [0,|\mathcal{Z}|]$  the following holds,
$$
V ^ {\pi *} (b _ {t}) \leq \mathrm {U D B} _ {n} ^ {\pi} (b _ {t}) = \bar {V} _ {n} ^ {\pi} (b _ {t}) + \epsilon_ {n, z} ^ {\pi} (b _ {t}). \tag {69}
$$
Additionally, from theorem 3 we have that,
$$
\left| V ^ {\pi} \left(b _ {t}\right) - \bar {V} _ {n} ^ {\pi} \left(b _ {t}\right) \right| \leq \epsilon_ {n, z} ^ {\pi} \left(b _ {t}\right), \tag {70}
$$
for any policy  $\pi$  and subset  $\overline{\mathcal{Z}}_n\subseteq \mathcal{Z}$  , thus,
$$
\bar {V} _ {n} ^ {\pi} \left(b _ {t}\right) - \epsilon_ {n, z} ^ {\pi} \left(b _ {t}\right) \leq V ^ {\pi} \left(b _ {t}\right) \leq V ^ {\pi *} \left(b _ {t}\right) \leq \bar {V} _ {n} ^ {\pi} \left(b _ {t}\right) + \epsilon_ {n, z} ^ {\pi} \left(b _ {t}\right). \tag {71}
$$
Since  $\epsilon_{n,z}^{\pi}(b_t) \to 0$  as  $n \to |\mathcal{Z}|$ , and  $|\mathcal{Z}|$  is finite, it is guaranteed that  $\mathrm{UDB}_n^\pi(b_t) \xrightarrow{n \to |\mathcal{Z}|} V^{\pi^*}(b_t)$  which completes our proof.
Moreover, depending on the algorithm implementation, the number of iterations can be finite (e.g. by directly choosing actions and observations to minimize the bound). A stopping criteria can also be verified by calculating the difference between the upper and lower bounds. The optimal solution is obtained once the upper bound equals the lower bound.
8.1 POMDP scenarios
We begin with a brief description of the Partially Observable Markov Decision Process (POMDP) scenarios implemented for the experiments. Each scenario was bounded by a finite number of time steps used for every episode, where each action taken by the agent led to a decrement in the number of time steps left. After the allowable time steps ended, the simulation was reset to its initial state.
8.1.1 Tiger POMDP
The Tiger is a classic POMDP problem Kaelbling et al. [1998], involves an agent making decisions between two doors, one concealing a tiger and the other a reward. The agent needs to choose among three actions, either open each one of the doors or listen to receive an observation about the tiger position. In our experiments, the POMDP was limited horizon of 5 steps. The problem consists of 3 actions, 2 observations and 2 states.
8.1.2 Discrete Light Dark
Is an adaptation from Sunberg and Kochenderfer [2018]. In this setting the agent needs to travel on a 1D grid to reach a target location. The grid is divided into a dark region, which offers noisy observations, and a light region, which offers accurate localization observations. The agent receives a penalty for every step and a reward for reaching the target location. The key challenge is to balance between information gathering by traveling towards the light area, and moving towards the goal region.
8.1.3 Laser Tag POMDP
In the Laser Tag problem, Somani et al. [2013], an agent has to navigate through a grid world, shoot and tag opponents by using a laser gun. The main goal is to tag as many opponents as possible within a given time frame. The grid is segmented into various sections that have varying visibility, characterized by obstacles that block the line of sight, and open areas. There are five possible actions,
moving in four cardinal directions (North, South, East, West) and shooting the laser. The observation space cardinality is  $|\mathcal{Z}| \approx 1.5 \times 10^6$ , which is described as a discretized normal distribution and reflect the distance measured by the laser. The states reflect the agent's current position and the opponents' positions. The agent receives a reward for tagging an opponent and a penalty for every movement, encouraging the agent to make strategic moves and shots.
8.1.4 Baby POMDP
The Baby POMDP is a classic problem that represents the scenario of a baby and a caregiver. The agent, playing the role of the caregiver, needs to infer the baby's needs based on its state, which can be either crying or quiet. The states in this problem represent the baby's needs, which could be hunger, discomfort or no need. The agent has three actions to choose from: feeding, changing the diaper, or doing nothing. The observations are binary, either the baby is crying or not. The crying observation does not uniquely identify the baby's state, as the baby may cry due to hunger or discomfort, which makes this a partially observable problem. The agent receives a reward when it correctly addresses the baby's needs and a penalty when the wrong action is taken.
8.2 Hyperparameters
The hyperparameters for both DB-DESPOT and AR-DESPOT algorithms were selected through a grid search. We explored an array of parameters for AR-DESPOT, choosing the highest-performing configuration. Specifically, the hyperparameter  $K$  was varied across  $\{10, 50, 500, 5000\}$ , while  $\lambda$  was evaluated at  $\{0, 0.01, 0.1\}$ . Similarly, DB-POMCP and POMCP were examined three different values for the exploration-exploitation weight,  $c = \{0.1, 1.0, 10.0\}$  multiplied by  $V_{max}$ , which denotes an upper bound for the value function.
For the initialization of the upper and lower bounds used by the algorithms, we used the maximal reward, multiplied by the remaining time steps of the episode,  $\mathcal{R}_{\mathrm{max}}\cdot (\mathcal{T} - t - 1)$
Finally, we provide our algorithm implementation in https://github.com/moranbar/Online-POMDP-Planning-with-Anytime-Deterministic-Guarantees.