SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Similar to Theorem 3,  $\epsilon_{x,z}^{\pi}(b_0)$  only contains probability values of instances from the simplified sets. However, this bound accounts for both the states and observations that are within the simplified spaces,  $x, z \in \bar{\mathcal{X}}, \bar{\mathcal{Z}}$ , which makes it a viable to compute at planning time. In contrast to Theorem 3, this bound can only be calculated at the root since it relies on the knowledge of the actual probability value of the prior,  $b_0(x)$ , for states  $x \in \bar{\mathcal{X}}$ , which are only available at the root. Furthermore, since now the belief density function at different belief nodes is not required, we demonstrate in the next section that a belief update can be avoided completely, which makes it suitable to attach guarantees to state-of-the-art algorithms.
The process of maintaining an upper bound for action-value function follows similarly to the one presented in the observation-only simplification subsection 3.1,
$$
\epsilon_ {x, z} ^ {\pi} \left(b _ {t}, a _ {t}\right) \triangleq \mathcal {R} _ {\max } \left[ 1 - \sum_ {x _ {t: t + 1}} b \left(x _ {t}\right) \bar {\mathbb {P}} \left(x _ {t + 1} \mid x _ {t}, a _ {t}\right) \right] + \tag {17}
$$
$$
\mathcal {R} _ {\max } \sum_ {\tau = t + 1} ^ {T} \left[ 1 - \sum_ {z _ {t + 1: \tau}} \sum_ {x _ {t: \tau}} b (x _ {t}) \overline {{\mathbb {P}}} (z _ {t + 1} | x _ {t + 1}) \overline {{\mathbb {P}}} (x _ {t + 1} | x _ {t}, a _ {t}) \prod_ {k = t + 2} ^ {\tau} \overline {{\mathbb {P}}} (z _ {k} | x _ {k}) \overline {{\mathbb {P}}} (x _ {k} | x _ {k - 1}, \pi_ {k - 1}) \right].
$$
In  $\epsilon_{x,z}^{\pi}(b_t, a_t)$ , the subscript of  $(x, z)$  differentiates the bound of the state- and observation-simplification from the observation-only simplification bound.
Computing the bounds in practice is straightforward and can be done iteratively, which fits many online planning algorithms. As it requires the summation over probability values for every considered trajectory,  $x_0, a_0, x_1, z_1, \ldots, a_{t-1}, x_t, z_t$ , as shown in equations (17) and (11). In the subsequent section, we provide further details and illustrate how the bounds derived from the state-observation simplification can be used to derive deterministic guarantees on policies obtained from existing state-of-the-art algorithms.
Algorithm 1 ALGORITHM-  $\mathcal{A}$  ..
function SEARCH function SIMULATE  $(h,d,\tau_d,\bar{\mathbb{P}}_d)$    
1: while time permits do 1: if  $d = 0$  then   
2: Generate states  $x$  from  $b_{0}$  . 2: return  $\delta (h)\gets 0$    
3:  $\tau_0\gets x$  3: end if   
4:  $\bar{\mathbb{P}}_0\gets b(x_0|h_0)$  4: Select action a.   
5: if  $\tau_0\notin \tau (h)$  then 5: Generate next states and observations,  $x^{\prime},z$    
6:  $\bar{\mathbb{P}} (h)\leftarrow \bar{\mathbb{P}} (h) + \bar{\mathbb{P}}_0$  6:  $\tau_d,\bar{\mathbb{P}}_\tau \gets \mathrm{FWDUPDATE}(ha,haz,\tau_d,\bar{\mathbb{P}}_\tau ,x')$    
7: end if   
8: SIMULATE  $(h,D,\tau_0,\bar{\mathbb{P}}_0)$    
9: end while   
10: return function FWDUPDATE(ha,haz,  $\tau_d,\bar{\mathbb{P}}_\tau ,x'$  10: return function BWDUPDATE(h,ha,d)   
1: if  $\tau_d\notin \tau (ha)$  then 1:  $\epsilon (ha) = \gamma^{D - d}V_{\max}(\bar{\mathbb{P}} (h) - \bar{\mathbb{P}} (ha)) + \gamma^{D - d - 1}$  Vmax(P(ha)-∑ P(haz))   
3: R(ha)R(ha)+r(x,a)   
4: end if   
5:  $\tau_d\gets \tau_d\cup \{x'\}$  2: U(ha)=R(ha)+y∑z|haU(haz)+e(ha)   
6:  $\bar{\mathbb{P}}_{\tau}\gets \bar{\mathbb{P}}_{\tau}\cdot Z_{z|x'}\cdot T_{x'|x,a}$  4: U(h)  $\leftarrow \max_{a'}\{U(ha')\}$    
7: if  $\tau_d\notin \tau (haz)$  then 5: L(h)  $\leftarrow \max_{a'}\{L(ha')\}$    
9:  $\tau (haz)\gets \tau (haz)\cup \{\tau_d\}$    
10: end if   
11: return
In this section we aim to describe how to fit our bounds to a general structure algorithm, named ALGORITHM -  $\mathcal{A}$ , which serves as an abstraction to many existing algorithms. To compute the deterministic bounds, we utilize Bellman's update and optimality criteria. This approach naturally fits dynamic programming approaches such as DESPOT [Ye et al., 2017] and AdaOPS [Wu et al., 2021]. However, it may also be attached with algorithms that rely on Monte-Carlo estimation, such as POMCP [Silver and Veness, 2010], by viewing the search tree as a policy tree.
While the analysis presented in section 3 is general and independent of the selection mechanism of the states or observations, we focus on sampling as a way to choose the simplified states at each belief node and the observations to branch from. Furthermore, the selection of the subspaces  $\bar{\mathcal{X}},\bar{\mathcal{Z}}$  need not be fixed, and may change over the course of time, similar to state-of-the-art algorithms, such as Hoerger and Kurniawati [2021], Silver and Veness [2010], Somani et al. [2013], Sunberg and Kochenderfer [2018], Wu et al. [2021]. Alternative selection methods may also be feasible, as sampling from the correct distribution is not required for the bounds to hold. Importantly, attaching our bounds to arbitrary exploration mechanism, such as in POMCP or DESPOT, leverages the derivations and convergence guarantees shown in 3.2. Clearly, the deterministic bounds remain valid, but a convergence analysis depends on the characteristics of the specific ALGORITHM -  $\mathcal{A}$  being used.
ALGORITHM -  $\mathcal{A}$  is outlined in algorithm 1. For the clarity of exposition, we assume the following; at each iteration a single state particle is propagated from the root node to the leaf (line 2 of function SEARCH). The selection of the next state and observations are done by sampling from the observation and transition models (line 5), and each iteration ends with the full horizon of the POMDP (lines 2). However, none of these are a restriction of our approach and may be replaced with arbitrary number of particles, arbitrary state and observation selection mechanism and a single or multiple expansions of new belief nodes at each iteration.
To compute the UDB value, we require both the state trajectory, denoted as  $\tau$ , and its probability value,  $\mathbb{P}_{\tau}$ . We use the state trajectory as a mechanism to avoid duplicate summation of an already accounted for probability value and is utilized to ascertain its uniqueness at a belief node. The probability value,  $\mathbb{P}_{\tau}$ , is the likelihood of visiting a trajectory  $\tau = \{x_0, a_0, x_1, z_1, \ldots, a_{t-1}, x_t, z_t\}$  and is calculated as the product of the prior, transition and observation likelihoods (line 6). If a trajectory was not seen before in a belief node, its reward value is multiplied by the trajectory likelihood, shown in 3. Each node maintains a cumulative sum of the likelihoods of all visited trajectories. This is then being
used to compute the upper and lower bounds, shown in lines 2. The bounds are computed in lines 1 represent the loss of holding only a subset of the sates in node  $ha$  from the set in node  $h$ , plus the loss of having only a subset of the future states and observations, where  $V_{\mathrm{max}}$  represent the maximum possible value function. A simple bound on the value function may be  $V_{\mathrm{max}} = \mathcal{R}_{\mathrm{max}} \cdot (D - d - 1)$  but other more sophisticated bounds are also possible, as well as different values for lower and upper bounds.
The time complexity for each posterior node, primarily depends on the specific algorithm being used. In the case of dynamic programming methods, such as DESPOT and AdaOPS, there is a negligible added computational complexity detailed below. In the case of Monte Carlo methods, such as POMCP, the computational complexity is  $O(|\mathcal{A}|)$  attributed mainly to the action-selection, while our approach adds another linear time complexity term, making it  $O(|\mathcal{A}| + |\bar{\mathcal{Z}}|)$  due to the summation over the simplified observation space. During each iteration of the algorithm, an "IF" statement is used to determine whether a specific trajectory has already been encountered at the current node. This verification process can potentially result in an added linear complexity of  $O(D)$ , where  $D$  represents the planning horizon. However, this overhead can be circumvented by assigning a unique ID value to each trajectory at the previous step and subsequently checking whether a pair, comprising the ID value and the new state, has already been visited. This approach reduces the overhead to an average time complexity of  $O(1)$  by utilizing hash maps efficiently.
In this section, we present the experimental results obtained by integrating deterministic bounds into a state-of-the-art algorithms namely, AR-DESPOT Somani et al. [2013] and POMCP Silver and Veness [2010] as a baseline. The primary objective of these experiments is to demonstrate the validity of our derived bounds, as presented in Theorem 2, and the corresponding algorithm outlined in Algorithm 1.
The implementation of our algorithm was carried out using the Julia programming language and evaluated through the Julia POMDPs package, provided by Egorov et al. [2017]. Although the library primarily focused on infinite horizon problems, we made the required modifications to extend its capabilities to accommodate finite-horizon POMDPs. The experiments were conducted on a computing platform consisting of an Intel(R) Core(TM) i7-7700 processor with 8 CPUs operating at  $3.60\mathrm{GHz}$  and  $15.6\mathrm{GHz}$ . The
Figure 2: The graph illustrates the convergence of the deterministic bounds using Deterministically-Bounded-AR-DESPOT, with upper and lower bounds depicted alongside the optimal value obtained through exhaustive search.
selection of hyper-parameters for the POMCP and AR-DESPOT solvers, and further details about the POMDPs used for our experiments are detailed in the appendix.
In Figure 2, we demonstrate the empirical convergence of the deterministic bounds in relation to planning time. For these experiments, we focused on a toy example, Tiger POMDP Kaelbling et al. [1998]. By conducting an exhaustive search and computing a full posterior belief for each node, we derived the optimal value, depicted as a dashed green line in figure 2. The graph presents the convergence of the bounds calculated with Deterministically-Bounded AR-DESPOT (DB-DESPOT), to the optimal value. The mean values of the upper and lower bounds across 100 simulations are plotted, accompanied by the standard deviation, for time increments of  $\Delta t = 0.1$  seconds.
Algorithms that provide probabilistic guarantees have a non-zero probability of taking a suboptimal action regardless of the planning time. In contrast, when given sufficient planning time, the deterministic bounds can certify the optimal action once the lower bound of an immediate action exceeds the upper bounds of all other actions. In our experiments, we observed that although the baseline algorithms, AR-DESPOT and POMCP, and the deterministically bounded algorithms, DB-DESPOT
and DB-POMCP, had access to the same tree and samples, AR-DESPOT and POMCP occasionally made incorrect decisions, resulting in degraded performance.
We evaluated the performance of both algorithms on different POMDPs, including the Tiger POMDP, Discrete Light Dark Sunberg and Kochenderfer [2018] and Baby POMDP. The corresponding results are summarized in Table 1. After each planning session the calculated best action is executed, the belief is updated according to the captured observation and a new planning session is invoked. The table reports the empirical mean and std of the cumulative rewards obtained by executing the calculated best actions based on 100 runs.
In the POMDPs with manageable state, action and observation sizes, both the DB-DESPOT and DB-POMCP algorithms frequently identified the optimal action within a 1-second time budget. It's important to highlight that these algorithms employ stochastic methods for belief tree exploration as in the baseline algorithms. Consequently, the outcomes between DB-DESPOT and DB-POMCP can vary, and there is no guarantee of consistently identifying the optimal action in every constrained-time planning session. An optimal action is determined when a lower bound of a given action is higher than any other upper bound; notably, this does not necessitates a construction of the entire planning tree and can be used as an early stopping mechanism for the optimal action. In the larger Laser Tag POMDP Somani et al. [2013], the DB-DESPOT and DB-POMCP did not outperform AR-DESPOT and POMCP. This discrepancy occurred because the planning time was insufficient to find an optimal action due to larger state, action and observation spaces of Laser Tag POMDP. While our proposed algorithms show promise in more compact POMDPs, their performance in larger-scale problems, like the Laser Tag POMDP, especially when constrained by external time budgets, merits further investigation.
Table 1: Performance comparison with and without deterministic bounds.
Algorithm	Tiger POMDP	Laser Tag	Discrete Light Dark	Baby POMDP
DB-DESPOT (ours)	3.74±0.48	-5.29±0.14	-5.29±0.01	-3.92±0.56
AR-DESPOT	2.82±0.55	-5.10±0.14	-61.53±5.80	-5.40±0.85
DB-POMCP (ours)	3.01±0.21	-3.97±0.24	-3.70±0.82	-4.48±0.57
POMCP	2.18±0.76	-3.92±0.27	-4.51±1.15	-5.39±0.63
In this work, we presented a novel methodology aimed at offering anytime, deterministic guarantees for approximate POMDP solvers. These solvers strategically leverage a subset of the state and observation spaces to alleviate the computational overhead. Our key proposition elucidates a linkage between the optimal value function, which is inherently computationally intensive, and a more tractable approximation frequently employed in contemporary algorithms. In the first part of the paper, we derived the theoretical relationship between the use of a selective subset of observations in a planning tree. One contribution of this work is the formulation of an upper deterministic bound (UDB) which governs exploration within the belief space, and is theoretically guaranteed to converge to the optimal value. This approach, however, depends on a complete belief update at each node of the tree, a requirement that can be computationally infeasible in many practical POMDPs. In the second part, we address this computational challenge, by extending our derivations to account for both a subset of the observations and states. This extension increases the feasibility of our approach and allows it to be incorporated into existing state-of-the-art algorithms. We have outlined how our methodology can be integrated within these algorithms. To illustrate the practical utility of our derivations, we applied them to certify and improve the solution of AR-DESPOT and POMCP algorithms, a state-of-the-art solvers for POMDPs.
While the application of our deterministic bounds within the context of the AR-DESPOT algorithm showcased its potential, there are several challenges that need to be addressed in future work. Firstly, the convergence rate of our proposed method remains an area for further exploration. A detailed theoretical analysis on this aspect could provide valuable insights into the practicality of our approach
for larger, more complex domains. Secondly, the current use of the UDB is predominantly restricted to problems characterized by low state dimensionality. Extending the UDB to handle higher state dimensionality is a significant challenge due to the increased computational complexity involved. Finally, we consider the integration of the UDB with simplified state and observation spaces to be a promising future direction. This could potentially lead to a more comprehensive and efficient strategy for handling larger POMDPs, thereby further enhancing the applicability of our deterministic approach.
This work was supported by the Israel Science Foundation (ISF) and by US NSF/US-Israel BSF.
Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47(2):235-256, 2002.
Maxim Egorov, Zachary N. Sunberg, Edward Balaban, Tim A. Wheeler, Jayesh K. Gupta, and Mykel J. Kochenderfer. POMDPs.jl: A framework for sequential decision making under uncertainty. Journal of Machine Learning Research, 18(26):1-5, 2017. URL http://jmlr.org/papers/v18/16-300.html.
Marcus Hoerger and Hanna Kurniawati. An on-line pomdp solver for continuous observation spaces. In IEEE Intl. Conf. on Robotics and Automation (ICRA), pages 7643-7649. IEEE, 2021.
L. P. Kaelbling, M. L. Littman, and A. R. Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1):99-134, 1998.
Michael H. Lim, Claire Tomlin, and Zachary N. Sunberg. Sparse tree search optimality guarantees in pomdpss with continuous observation spaces. In Intl. Joint Conf. on AI (IJCAI), pages 4135-4142, 7 2020.
Michael H Lim, Tyler J Becker, Mykel J Kochenderfer, Claire J Tomlin, and Zachary N Sunberg. Generalized optimality guarantees for solving continuous observation pomdps through particle belief mdp approximation. arXiv preprint arXiv:2210.05015, 2022.
David Silver and Joel Veness. Monte-carlo planning in large pomdpds. In Advances in Neural Information Processing Systems (NIPS), pages 2164-2172, 2010.
Adhiraj Somani, Nan Ye, David Hsu, and Wee Sun Lee. Despot: Online pomdp planning with regularization. In NIPS, volume 13, pages 1772-1780, 2013.
Zachary Sunberg and Mykel Kochenderfer. Online algorithms for pomdps with continuous state, action, and observation spaces. In Proceedings of the International Conference on Automated Planning and Scheduling, volume 28, 2018.
Chenyang Wu, Guoyu Yang, Zongzhang Zhang, Yang Yu, Dong Li, Wulong Liu, and Jianye Hao. Adaptive online packing-guided search for pomdps. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems (NIPS), volume 34, pages 28419-28430. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper_files/paper/2021/file/ef41d488755367316f04fc0e0e9dc9fc-Paper.pdf.
Nan Ye, Adhiraj Somani, David Hsu, and Wee Sun Lee. Despot: Online pomdp planning with regularization. JAIR, 58:231-266, 2017.
7 Mathematical Analysis
We start by restating the definition of the simplified value function,
$$
\begin{array}{l} \bar {V} ^ {\pi} \left(\bar {b} _ {t}\right) \triangleq r \left(\bar {b} _ {t}, \pi_ {t}\right) + \bar {\mathbb {E}} \left[ \bar {V} \left(b _ {t}\right) \right] (18) \\ = \sum_ {x _ {t}} \bar {b} \left(x _ {t}\right) r \left(x _ {t}, \pi_ {t}\right) + \sum_ {z _ {t}} \bar {\mathbb {P}} \left(z _ {t + 1} \mid H _ {t + 1} ^ {-}\right) \bar {V} (\bar {b} \left(z _ {t + 1}\right)), (19) \\ \end{array}
$$
7.1 Theorem 1
Theorem 3. Let  $b_{t}$  belief state at time  $t$ , and  $T$  be the last time step of the POMDP. Let  $V^{\pi}(b_{t})$  be the theoretical value function by following a policy  $\pi$ , and let  $\bar{V}^{\pi}(b_{t})$  be the simplified value function, as defined in (18), by following the same policy. Then, for any policy  $\pi$ , the difference between the theoretical and simplified value functions is bounded as follows,
$$
\left| V ^ {\pi} \left(b _ {t}\right) - \bar {V} ^ {\pi} \left(b _ {t}\right) \right| \leq \mathcal {R} _ {\max } \sum_ {\tau = t + 1} ^ {T} \left[ 1 - \sum_ {z _ {t + 1: \tau}} \sum_ {x _ {t: \tau}} b \left(x _ {t}\right) \prod_ {k = t + 1} ^ {\tau} \overline {{\mathbb {P}}} \left(z _ {k} \mid x _ {k}\right) \mathbb {P} \left(x _ {k} \mid x _ {k - 1}, \pi_ {k - 1}\right) \right] \triangleq \epsilon_ {z} ^ {\pi} \left(b _ {t}\right). \tag {20}
$$
Proof. For notational convenience, we derive the bounds for the value function by denoting the prior belief as  $b_{0}$ ,
$$
V _ {0} ^ {\pi} \left(b _ {0}\right) = \mathbb {E} _ {z _ {1: T}} \left[ \sum_ {t = 0} ^ {T} r \left(b _ {t}, a _ {t}\right) \right] \tag {21}
$$
applying the belief update equation,
$$
\begin{array}{l} V _ {0} ^ {\pi} \left(b _ {0}\right) = \sum_ {z _ {1: T}} \prod_ {\tau = 1} ^ {T} \mathbb {P} \left(z _ {\tau} \mid H _ {\tau} ^ {-}\right) \sum_ {t = 0} ^ {T} \left[ \sum_ {x _ {t}} \frac {\mathbb {P} \left(z _ {t} \mid x _ {t}\right) \sum_ {x _ {t - 1}} \mathbb {P} \left(x _ {t} \mid x _ {t - 1} , \pi_ {t - 1}\right) b _ {t - 1}}{\mathbb {P} \left(z _ {t} \mid H _ {t} ^ {-}\right)} r \left(x _ {t}, a _ {t}\right) \right] (22) \\ = \sum_ {z _ {1: T}} \prod_ {\tau = 1} ^ {T} \mathbb {P} \left(z _ {\tau} \mid H _ {\tau} ^ {-}\right) \sum_ {t = 0} ^ {T} \left[ \sum_ {x _ {0: t}} \frac {\prod_ {k = 1} ^ {t} \mathbb {P} \left(z _ {k} \mid x _ {k}\right) \mathbb {P} \left(x _ {k} \mid x _ {k - 1} , \pi_ {k - 1}\right) b \left(x _ {0}\right)}{\prod_ {\tau = 1} ^ {t} \mathbb {P} \left(z _ {\tau} \mid H _ {\tau} ^ {-}\right)} r \left(x _ {t}, a _ {t}\right) \right] (23) \\ = \sum_ {t = 0} ^ {T} \sum_ {z _ {1: T}} \sum_ {x _ {0: T}} \prod_ {k = 1} ^ {t} \mathbb {P} \left(z _ {k} \mid x _ {k}\right) \mathbb {P} \left(x _ {k} \mid x _ {k - 1}, \pi_ {k - 1}\right) b \left(x _ {0}\right) r \left(x _ {t}, a _ {t}\right) (24) \\ \end{array}
$$
which applies similarly to the simplified value function,
$$
\bar {V} _ {0} ^ {\pi} \left(b _ {0}\right) = \sum_ {t = 0} ^ {T} \sum_ {z _ {1: T}} \sum_ {x _ {0: T}} \prod_ {k = 1} ^ {t} \bar {\mathbb {P}} \left(z _ {k} \mid x _ {k}\right) \mathbb {P} \left(x _ {k} \mid x _ {k - 1}, \pi_ {k - 1}\right) b \left(x _ {0}\right) r \left(x _ {t}, a _ {t}\right). \tag {25}
$$
We begin the derivation by focusing on a single time step,  $t$ , and later generalize to the complete value function.
$$
\begin{array}{l} \left| \mathbb {E} _ {z _ {1: t}} \left[ r \left(b _ {t}\right) \right] - \overline {{\mathbb {E}}} _ {z _ {1: t}} \left[ r \left(\bar {b} _ {t}\right) \right] \right| (26) \\ = \left| \sum_ {z _ {1: t}} \sum_ {x _ {0: t}} \left[ \prod_ {k = 1} ^ {t} \mathbb {P} \left(z _ {k} \mid x _ {k}\right) \mathbb {P} \left(x _ {k} \mid x _ {k - 1}, \pi_ {k - 1}\right) b \left(x _ {0}\right) r \left(x _ {t}\right) - \prod_ {k ^ {\prime} = 1} ^ {t} \overline {{\mathbb {P}}} \left(z _ {k ^ {\prime}} \mid x _ {k ^ {\prime}}\right) \mathbb {P} \left(x _ {k ^ {\prime}} \mid x _ {k ^ {\prime} - 1}, \pi_ {k ^ {\prime} - 1}\right) b \left(x _ {0}\right) r \left(x _ {t}\right) \right] \right| (27) \\ \leq \sum_ {z _ {1: t}} \sum_ {x _ {0: t}} \left| r \left(x _ {t}\right) \left[ \prod_ {k = 1} ^ {t} \mathbb {P} \left(z _ {k} \mid x _ {k}\right) \mathbb {P} \left(x _ {k} \mid x _ {k - 1}, \pi_ {k - 1}\right) b \left(x _ {0}\right) - \prod_ {k ^ {\prime} = 1} ^ {t} b \left(x _ {0}\right) \overline {{\mathbb {P}}} \left(z _ {k ^ {\prime}} \mid x _ {k ^ {\prime}}\right) \mathbb {P} \left(x _ {k ^ {\prime}} \mid x _ {k ^ {\prime} - 1}, \pi_ {k ^ {\prime} - 1}\right) \right] \right. (28) \\ = \sum_ {z _ {1: t}} \sum_ {x _ {0: t}} | r (x _ {t}) | \left[ \prod_ {k = 1} ^ {t} \mathbb {P} \left(z _ {k} \mid x _ {k}\right) \mathbb {P} \left(x _ {k} \mid x _ {k - 1}, \pi_ {k - 1}\right) b \left(x _ {0}\right) - \prod_ {k ^ {\prime} = 1} ^ {t} b \left(x _ {0}\right) \overline {{\mathbb {P}}} \left(z _ {k ^ {\prime}} \mid x _ {k ^ {\prime}}\right) \mathbb {P} \left(x _ {k ^ {\prime}} \mid x _ {k ^ {\prime} - 1}, \pi_ {k ^ {\prime} - 1}\right) \right] (29) \\ \end{array}
$$
where the second transition is due to triangle inequality, the third transition is equality by the construction, i.e. using the simplified observation models imply that the difference is nonnegative. We add and subtract, followed by rearranging terms,
$$
= \sum_ {z _ {1: t}} \sum_ {x _ {0: t}} | r (x _ {t}) | \tag {30}
$$