SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 1: CMLO performance curves for different threshold settings over different random seeds, where 3.0 is the threshold recommended in the paper.
5.2 Comparison with baselines
To highlight our algorithm, we compare USB-PO against some SOTA MBRL and MFRL algorithms. The MBRL baselines include CMLO [21], which carefully chooses the threshold for different environments to constrain the impacts of model shift, ALM [14], which learns the representation, model, and policy into one objective, PDML [50], which uses historical policy sequence to aim the model training process, MBPO [20], which is the variant without our proposed model fine-tuning process and uses return discrepancy to get a performance improvement guarantee. The MFRL baselines include SAC [15], the SOTA MFRL algorithm in terms of asymptotic performance and PPO [40], which explores monotonic improvement under the model-free setting.
We evaluate USB-PO and these baselines on six MuJoCo [46] continuous control tasks in OpenAI Gym [3], covering Humanoid, Walker2d, Ant, HalfCheetah, Hopper and Inverted-Pendulum, with more details showing in the appendix. To be fair, we employ the standard 1000-step version of these tasks with the same environment settings.
Figure 2 shows the learning curves of all compared methods, together with the asymptotic performance. The results show that our algorithm is remarkably advanced over the MFRL algorithms
Figure 2: Comparison against baselines on continuous control benchmarks. Solid curves refer to the mean performance of trials over different random seeds, and shaded area refers to the standard deviation of these trials. Dashed lines refer to the asymptotic performance of SAC (at 3M steps). Note that PDML is not evaluated in InvertedPendulum-v2.
regarding sample efficiency, along with asymptotic performance on par with the SOTA MFRL algorithm SAC. Compared to the MBRL baselines, our method achieves higher sample efficiency and better final performance. Notably, compared to CMLO, which requires a finely chosen threshold for each environment to constrain the impacts of model shift, our method utilizes the same learning rate for the fine-tuning process in all environments. This further validates unifying the model shift and the model bias to adjust the model updates adaptively is reasonable.
Computational Cost. We report our computational cost compared to MBPO [20] in Appendix D.4. Although USB-PO is a two-phase model training process, continuing to use the fine-tuned model for the next iteration has the potential to accelerate model convergence and then possibly reduce the training time.
5.3 How to understand USB-PO
In this section, we first design an experiment to illustrate the value magnitude of  $\Delta$ , the model shift term and the model bias term, validating whether using the second-order Wasserstein distance satisfies the prerequisites for getting a performance improvement guarantee, and then use more in-depth experiments to illustrate how USB-PO works and show its superiority.
Value Magnitude. We choose 2 challenging tasks in MuJoCo, HalfCheetah and Walker2d, and plot the value of  $\Delta$ , the model shift term and the model bias term during the training process respectively. To approximate the  $\Delta$  practically, we use the samples from the model replay
buffer before the update of the model and the policy to approximate sampling from  $d_{M_1}^{\pi_1}$  and use the samples generated by them after update to approximate sampling from  $d_{M_2}^{\pi_2}$ .
As shown in Figure 3, the magnitude of  $\Delta$  generally oscillates in a slight manner around 0 while the magnitude of model shift term and the model bias term are much larger than  $\Delta$  even in the converged cases. Therefore, it is reasonable to ignore the impacts of  $\Delta$  in Eq.(8). This further supports that minimizing Eq.(3) can get a performance improvement guarantee.
Figure 3: The value magnitude of the  $\Delta$ , the model shift term and the model bias term during the training process in HalfCheetah and Walker2d.
Working Mechanism. To illustrate the significance of our method, we calculate the optimization objective value before and after the fine-tuning process and plot their difference, as shown in Figure 4. All random seeds show the consistent result that at the beginning of training the sum of model bias and model shift is large and thus the fine-tuning magnitude is relatively large. As the model is trained to converge, the fine-tuning process also gradually converges to ensure the stability of the model. Notice that in some cases the fine-tuning process does not choose to fine-tune the model updates actually (0 refers to no update, which means the updates in phase 1 are reasonable), which further supports our theorem and the motivation of this paper, i.e., actively adaptive adjustment of the model updates to get a performance improvement guarantee, rather than passively waiting for model shift to surpass a given threshold to make a substantial update [21]. Based on the performance comparison in Figure 2, we argue that actively adaptive updates are more beneficial.
Figure 4: (a) the difference of the optimization objective value before and after the fine-tuning process during the training process over 5 random seeds. (b) the average prediction error during the training process over these random seeds. The model is generally near convergence when training times reach 1K.
Figure 5: (a) we choose a specific random seed to show the details of the first 30 training times, covering the difference of optimization objective value, the model shift term and the model bias term before and after the fine-tuning process. (b) the difference of average prediction error before and after the fine-tuning process during the training process over the previously used random seeds.
We further plot the difference of the model shift term, the model bias term and the average prediction error before and after the fine-tuning process. As shown in Figure 5 (a), when the fine-tuning actually operates, the difference of the model shift term and the model bias term are both positive. As shown in Figure 5 (b), the fine-tuning process has a positive effect on the reduction of the average prediction error. These observations suggest that USB-PO has other positive effects during the fine-tuning process, i.e. potentially reducing both model shift and model bias and leading to model overfitting avoidance.
Figure 6: The average return of different optimization objective variants over different random seeds.
Figure 7: The average return of USB-PO and MBPO with different learning rates over different random seeds.
Additionally, the change in the adjustment magnitude of the average prediction error indicates that the model can converge under the role of the fine-tuning process, which is consistent with Figure 4, forming bi-verification.
To conclude, when USB-PO actively recognizes that an improper update happens, it performs a pullback operation (the model fine-tuning process) to secure the performance improvement guarantee while avoiding model overfitting. In contrast, when USB-PO considers that a reasonable update is done, no actual operation will be taken. According to the performance comparison results in Figure 2, it is verified that the model quality plays a determinant role in the Dyna-Q [44] algorithms.
Figure 8: Performance of different learning rates on Walker2d and HalfCheetah during the whole training process. Each learning rate is repeated several times over different random seeds and the dashed line refers to the maximum average returns of MBPO.
In this section, we design 3 ablation experiments to strengthen our superiority.
**Optimization Objective Variants.** We set three variants for our optimization objective, covering (1) without the model shift term and the model bias term, which is equal to MBPO [20], (2) with the model shift term, (3) with the model bias term. As shown in Figure 6, only optimizing the model shift term results in a drop in sample efficiency since the fine-tuning process makes  $M_2$  tend to update towards  $M_1$  and only optimizing the model bias term leads to performance deterioration due to excessive model updates.
Not Equivalent to Limiting the Update Magnitude. We set different learning rates for MBPO to compare with USB-PO. As Figure 7 shows, USB-PO is more dominant from the perspective of the average return, which further strengthens that USB-PO is not equivalent to limiting the update magnitude, but rather beneficial to get a performance improvement guarantee.
Learning Rate Performance Comparison. We set up an ablation experiment on the learning rate of the fine-tuning process. As shown in Figure 8, unlike CMLO which is strongly dependent on a carefully chosen threshold for each environment to constrain the impacts of model shift, USB-PO is less sensitive to the learning rate of phase 2.
In this paper, we propose a novel MBRL algorithm called USB-PO, which can unify model shift and model bias, enabling adaptive adjustment of their impacts to get a performance improvement guarantee. We further find that our method can potentially reduce both model shift and model bias, leading to model overfitting avoidance. Empirical results on several challenging benchmark tasks validate the superiority of our algorithm and more in-depth experiments are conducted to demonstrate our mechanism. The limitation of our work is we do not investigate the change of model shift and model bias in this optimization problem theoretically. Therefore, one direction that merits further research is how to solidly interpret the variation of model shift and model bias in this fine-tuning process. Further, we aim to propose a general algorithmic framework in this paper. Due to the simple design of our model architecture, we find it difficult to capture some complex and practical locomotion. We hope to combine our adaptive adjusting technique with more advanced model architectures in the future.
This work is supported by the National Key Research and Development Program of China (No. 2021YFB2501104 No. 2020YFA0711402). We thank Michael Janner, Raj Ghugare, Zifan Wu, and Xiyao Wang for providing the baseline results and thank Tianying Ji for the worthy discussions.
[1] Brandon Amos et al. "On the model-based stochastic value gradient for continuous reinforcement learning". In: Learning for Dynamics and Control. PMLR. 2021, pp. 6-20.
[2] Kavosh Asadi et al. "Combating the compounding-error problem with a multi-step model". In: arXiv preprint arXiv:1905.13320 (2019).
[3] Greg Brockman et al. "Openai gym". In: arXiv preprint arXiv:1606.01540 (2016).
[4] Jacob Buckman et al. "Sample-efficient reinforcement learning with stochastic ensemble value expansion". In: Advances in neural information processing systems 31 (2018).
[5] Annie Chen et al. “You Only Live Once: Single-Life Reinforcement Learning”. In: Advances in Neural Information Processing Systems 35 (2022), pp. 14784–14797.
[6] Kurtland Chua et al. "Deep reinforcement learning in a handful of trials using probabilistic dynamics models". In: Advances in neural information processing systems 31 (2018).
[7] Sebastian Curi, Felix Berkenkamp, and Andreas Krause. "Efficient model-based reinforcement learning through optimistic policy search and planning". In: Advances in Neural Information Processing Systems 33 (2020), pp. 14156-14170.
[8] Vladimir Feinberg et al. "Model-based value estimation for efficient model-free reinforcement learning". In: arXiv preprint arXiv:1803.00101 (2018).
[9] Shuo Feng et al. “Dense reinforcement learning for safety validation of autonomous vehicles”. In: Nature 615.7953 (2023), pp. 620–627.
[10] Lukas Froehlich et al. "On-Policy Model Errors in Reinforcement Learning". In: International Conference on Learning Representations. 2022. URL: https://openreview.net/forum?id=81elaeOt-sd.
[11] Scott Fujimoto, Herke Hoof, and David Meger. “Addressing function approximation error in actor-critic methods”. In: International conference on machine learning. PMLR. 2018, pp. 1587–1596.
[12] Yarin Gal, Jiri Hron, and Alex Kendall. "Concrete dropout". In: Advances in neural information processing systems 30 (2017).
A Proof Sketch
To better clarify our theoretical results, we provide a proof sketch here. Firstly, we decompose the performance difference bound under the model-based setting into three terms (Theorem 1). Secondly, by means of using Return Bound (Theorem 2), we can bound these three terms individually (Theorem 3). Then, we can do some transformation to get Unified Model Shift and Model Bias Bound (Theorem 4), which bounds the model shift term and the model bias term in total variation form. However, due to the intractable property of  $\Delta$ , we further explore the upper bound of  $|\Delta|$  (Theorem 5), finding that  $\Delta$  can be ignored. Finally, by the Integral Probability Metrics (Lemma 3) and the property of the Wasserstein distance, we derive the target which bounds the model shift term and the model bias term in the Wasserstein distance form.
Figure 9: Theoretical sketch of USB-PO.
In this section, we provide some proof to support our theoretical analysis.
Lemma 1 (Total variation Distance). Consider a measurable space  $(\Omega, \Sigma)$  and probability measures  $P$  and  $Q$  are defined on  $(\Omega, \Sigma)$ . The total variation distance between  $P$  and  $Q$  is defined as:
$$
D _ {T V} (P | | Q) = \sup  _ {A \in \Sigma} | P (A) - Q (A) | \tag {12}
$$
Eq.(12) can be equivalently written as:
$$
D _ {T V} (P | | Q) = \frac {1}{2} \sum_ {\omega \in \Omega} \left| P (\{\omega \}) - Q (\{\omega \}) \right| \tag {13}
$$
Proof: The proof of this lemma can be found in [26].
Lemma 2 (Total Variation Distance of Joint Distributions). Given two distributions  $p(x,y) = p(x)p(y|x)$  and  $q(x,y) = q(x)q(y|x)$ , the total variation distance between them can be bounded as:
$$
D _ {T V} (p (x, y) | | q (x, y)) \leq D _ {T V} (p (x) | | q (x)) + \max  _ {x} D _ {T V} (p (y | x) | | q (y | x)) \tag {14}
$$
$$
\begin{array}{l} D _ {T V} (p (x, y) | | q (x, y)) = \frac {1}{2} \sum_ {x, y} | p (x, y) - q (x, y) | \\ = \frac {1}{2} \sum_ {x, y} | p (x) p (y | x) - q (x) q (y | x) | \\ = \frac {1}{2} \sum_ {x, y} | p (x) p (y | x) - p (x) q (y | x) + p (x) q (y | x) - q (x) q (y | x) | \\ \leq \frac {1}{2} \sum_ {x, y} p (x) | p (y | x) - q (y | x) | + | p (x) - q (x) | q (y | x) \tag {15} \\ = \frac {1}{2} \sum_ {x, y} p (x) | p (y | x) - q (y | x) | + \frac {1}{2} \sum_ {x} | p (x) - q (x) | \\ = \mathbb {E} _ {x \sim p (x)} [ D _ {T V} (p (y | x) | | q (y | x)) ] + D _ {T V} (p (x) | | q (x)) \\ \leq D _ {T V} (p (x) | | q (x)) + \max  _ {x} D _ {T V} (p (y | x) | | q (y | x)) \\ \end{array}
$$
Lemma 3 (Integral Probability Metrics). Consider a measurable space  $(\mathcal{X},\Sigma)$ . The integral probability metric associated with a class  $\mathcal{F}$  of real-valued functions on  $\mathcal{X}$  is defined as
$$
d _ {\mathcal {F}} (P, Q) = \sup  _ {f \in \mathcal {F}} | \mathbb {E} _ {X \sim P} [ f (X) ] - \mathbb {E} _ {Y \sim Q} [ f (Y) ] | \tag {16}
$$
where  $P$  and  $Q$  are probability measures on  $\mathcal{X}$ . We demonstrate the following special cases:
(a) If  $\mathcal{F} = \{f:||f||_{\infty}\leq c\}$ , then we have
$$
d _ {\mathcal {F}} (P, Q) = c D _ {T V} (P | | Q) \tag {17}
$$
(b) If  $\mathcal{F}$  is the set of  $L-$  Lipschitz function with a norm  $\| \cdot \|$ , then we have
$$
d _ {\mathcal {F}} (P, Q) = L W _ {1} (P, Q) \tag {18}
$$
In our paper, to distinguish the dynamic transition function, we choose  $\mathcal{F}$  to be the class covering  $V_{M}^{\pi}$ . Since the value function can converge to  $\frac{r_{max}}{1 - \gamma}$ , it only needs to satisfy the  $L_{v}$ -Lipschitz continuity and thus we can get  $\frac{r_{max}}{1 - \gamma} D_{TV}(p_M||p_{M'}) = L_vW_1(p_M,p_{M'})$  for any arbitrary model  $M,M'$ .
C Missing Proof
$$
V ^ {\pi_ {2} \mid M _ {2}} - V ^ {\pi_ {1} \mid M _ {1}} = \left(V ^ {\pi_ {2} \mid M _ {2}} - V _ {M _ {2}} ^ {\pi_ {2}}\right) - \left(V ^ {\pi_ {1} \mid M _ {1}} - V _ {M _ {1}} ^ {\pi_ {1}}\right) + \left(V _ {M _ {2}} ^ {\pi_ {2}} - V _ {M _ {1}} ^ {\pi_ {1}}\right) \tag {19}
$$
Proof: We introduce two additional terms  $V_{M_1}^{\pi_1}$  and  $V_{M_2}^{\pi_2}$  that allow the performance difference bound objective to be divided into three operators based on the return bounds, which can be reformulated separately.
$$
\begin{array}{l} V ^ {\pi_ {2} \mid M _ {2}} - V ^ {\pi_ {1} \mid M _ {1}} = V ^ {\pi_ {2} \mid M _ {2}} - V ^ {\pi_ {1} \mid M _ {1}} + \left(V _ {M _ {1}} ^ {\pi_ {1}} - V _ {M _ {1}} ^ {\pi_ {1}}\right) + \left(V _ {M _ {2}} ^ {\pi_ {2}} - V _ {M _ {2}} ^ {\pi_ {2}}\right) \tag {20} \\ = \left(V ^ {\pi_ {2} | M _ {2}} - V _ {M _ {2}} ^ {\pi_ {2}}\right) - \left(V ^ {\pi_ {1} | M _ {1}} - V _ {M _ {1}} ^ {\pi_ {1}}\right) + \left(V _ {M _ {2}} ^ {\pi_ {2}} - V _ {M _ {1}} ^ {\pi_ {1}}\right) \\ \end{array}
$$
Theorem 2 (Return Bound). Let  $R_{max}$  denote the bound of the reward function,  $\epsilon_{\pi}$  denote  $\max_s D_{TV}(\pi_1||\pi_2)$  and  $\epsilon_{M_1}^{M_2}$  denote  $\mathbb{E}_{(s,a)\sim d_{M_1}^{\pi_1}}[D_{TV}(p_{M_1}||p_{M_2})]$ . For two arbitrary policies  $\pi_1,\pi_2\in \Pi$ , the expected return under two arbitrary models  $M_1,M_2\in \mathcal{M}$  can be bounded as,
$$
V _ {M _ {2}} ^ {\pi_ {2}} - V _ {M _ {1}} ^ {\pi_ {1}} \geq - 2 R _ {m a x} \left(\frac {\epsilon_ {\pi}}{(1 - \gamma) ^ {2}} + \frac {\gamma}{(1 - \gamma) ^ {2}} \epsilon_ {M _ {1}} ^ {M _ {2}}\right) \tag {21}
$$
Proof: We give the thorough proof referring to Lemma B.4 in MBPO [20] as follows.
$$
\begin{array}{l} V _ {M _ {2}} ^ {\pi_ {2}} - V _ {M _ {1}} ^ {\pi_ {1}} = \sum_ {t = 0} ^ {\infty} \gamma^ {t} \sum_ {s, a} \left(p _ {t, M _ {2}} ^ {\pi_ {2}} (s, a) - p _ {t, M _ {1}} ^ {\pi_ {1}} (s, a)\right) r (s, a) \\ \geq - R _ {m a x} \sum_ {t = 0} ^ {\infty} \gamma^ {t} \sum_ {s, a} \left| p _ {t, M _ {2}} ^ {\pi_ {2}} (s, a) - p _ {t, M _ {1}} ^ {\pi_ {1}} (s, a) \right| \tag {22} \\ = - 2 R _ {m a x} \sum_ {t = 0} ^ {\infty} \gamma^ {t} D _ {T V} (p _ {t, M _ {1}} ^ {\pi_ {1}} (s, a) | | p _ {t, M _ {2}} ^ {\pi_ {2}} (s, a)) \\ \end{array}
$$
According to the Lemma 2, we have:
$$
\begin{array}{l} D _ {T V} \left(p _ {t, M _ {1}} ^ {\pi_ {1}} (s, a) \mid \mid p _ {t, M _ {2}} ^ {\pi_ {2}} (s, a)\right) \leq D _ {T V} \left(p _ {t, M _ {1}} ^ {\pi_ {1}} (s) \mid \mid p _ {t, M _ {2}} ^ {\pi_ {2}} (s)\right) + \max  _ {s} D _ {T V} \left(\pi_ {1} (\cdot | s) \mid \mid \pi_ {2} (\cdot | s)\right) \tag {23} \\ = D _ {T V} \left(p _ {t, M _ {1}} ^ {\pi_ {1}} (s) | | p _ {t, M _ {2}} ^ {\pi_ {2}} (s)\right) + \epsilon_ {\pi} \\ \end{array}
$$
Further we expand the first term:
$$
\begin{array}{l} D _ {T V} \left(p _ {t, M _ {1}} ^ {\pi_ {1}} (s) \mid \mid p _ {t, M _ {2}} ^ {\pi_ {2}} (s)\right) \\ = \frac {1}{2} \sum_ {s} \left| p _ {t, M _ {1}} ^ {\pi_ {1}} (s) - p _ {t, M _ {2}} ^ {\pi_ {2}} (s) \right| \\ = \frac {1}{2} \sum_ {s} \left| \sum_ {s ^ {\prime}} p _ {M _ {1}} ^ {\pi_ {1}} (s | s ^ {\prime}) p _ {t - 1, M _ {1}} ^ {\pi_ {1}} (s ^ {\prime}) - p _ {M _ {2}} ^ {\pi_ {2}} (s | s ^ {\prime}) p _ {t - 1, M _ {2}} ^ {\pi_ {2}} (s ^ {\prime}) \right| \\ \leq \frac {1}{2} \sum_ {s} \sum_ {s ^ {\prime}} | p _ {M _ {1}} ^ {\pi_ {1}} (s | s ^ {\prime}) p _ {t - 1, M _ {1}} ^ {\pi_ {1}} (s ^ {\prime}) - p _ {M _ {2}} ^ {\pi_ {2}} (s | s ^ {\prime}) p _ {t - 1, M _ {2}} ^ {\pi_ {2}} (s ^ {\prime}) | \\ \leq \frac {1}{2} \sum_ {s, s ^ {\prime}} p _ {t - 1, M _ {1}} ^ {\pi_ {1}} (s ^ {\prime}) | p _ {M _ {1}} ^ {\pi_ {1}} (s | s ^ {\prime}) - p _ {M _ {2}} ^ {\pi_ {2}} (s | s ^ {\prime}) | + p _ {M _ {2}} ^ {\pi_ {2}} (s | s ^ {\prime}) | p _ {t - 1, M _ {1}} ^ {\pi_ {1}} (s ^ {\prime}) - p _ {t - 1, M _ {2}} ^ {\pi_ {2}} (s ^ {\prime}) | \\ = \frac {1}{2} \mathbb {E} _ {s ^ {\prime} \sim p _ {t - 1, M _ {1}} (s ^ {\prime})} [ \sum_ {s} | p _ {M _ {1}} ^ {\pi_ {1}} (s | s ^ {\prime}) - p _ {M _ {2}} ^ {\pi_ {2}} (s | s ^ {\prime}) | ] + D _ {T V} \big (p _ {t - 1, M _ {1}} ^ {\pi_ {1}} (s ^ {\prime}) | | p _ {t - 1, M _ {2}} ^ {\pi_ {2}} (s ^ {\prime}) \big) \\ = \frac {1}{2} \sum_ {t ^ {\prime} = 1} ^ {t} \mathbb {E} _ {s ^ {\prime} \sim p _ {t ^ {\prime} - 1, M _ {1}} ^ {\pi_ {1}} (s ^ {\prime})} \left[ \sum_ {s} \left| p _ {M _ {1}} ^ {\pi_ {1}} (s | s ^ {\prime}) - p _ {M _ {2}} ^ {\pi_ {2}} (s | s ^ {\prime}) \right| \right] \tag {24} \\ = \frac {1}{2} \sum_ {t ^ {\prime} = 1} ^ {t} \mathbb {E} _ {s ^ {\prime} \sim p _ {t ^ {\prime} - 1, M _ {1}} ^ {\pi_ {1}}} (s ^ {\prime}) [ \sum_ {s} | \sum_ {a} p _ {M _ {1}} ^ {\pi_ {1}} (s, a | s ^ {\prime}) - p _ {M _ {2}} ^ {\pi_ {2}} (s, a | s ^ {\prime}) | ] \\ \leq \frac {1}{2} \sum_ {t ^ {\prime} = 1} ^ {t} \mathbb {E} _ {s ^ {\prime} \sim p _ {t ^ {\prime} - 1, M _ {1}} ^ {\pi_ {1}} (s ^ {\prime})} [ \sum_ {s, a} | p _ {M _ {1}} ^ {\pi_ {1}} (s, a | s ^ {\prime}) - p _ {M _ {2}} ^ {\pi_ {2}} (s, a | s ^ {\prime}) | ] \\ = \sum_ {t ^ {\prime} = 1} ^ {t} \mathbb {E} _ {s ^ {\prime} \sim p _ {t ^ {\prime} - 1, M _ {1}} ^ {\pi_ {1}} (s ^ {\prime})} D _ {T V} \left(p _ {M _ {1}} ^ {\pi_ {1}} (s, a | s ^ {\prime}) \right\lvert   \left| p _ {M _ {2}} ^ {\pi_ {2}} (s, a | s ^ {\prime})\right) \\ \leq \sum_ {t ^ {\prime} = 1} ^ {t} \mathbb {E} _ {s ^ {\prime} \sim p _ {t ^ {\prime} - 1, M _ {1}} ^ {\pi_ {1}} (s ^ {\prime})} [ \epsilon_ {\pi} + \mathbb {E} _ {a \sim \pi_ {1}} [ D _ {T V} (p _ {M _ {1}} (s | s ^ {\prime}, a) | | p _ {M _ {2}} (s | s ^ {\prime}, a)) ] ] \\ = t \epsilon_ {\pi} + \sum_ {t ^ {\prime} = 1} ^ {t} \mathbb {E} _ {s ^ {\prime}, a \sim p _ {t ^ {\prime} - 1, M _ {1}} (s ^ {\prime}, a)} D _ {T V} \left(p _ {M _ {1}} (s | s ^ {\prime}, a) \right\lvert   \left| p _ {M _ {2}} (s | s ^ {\prime}, a)\right) \\ \end{array}
$$
Then move the result of Eq.(24) to Eq.(23), we can get:
$$
D _ {T V} \left(p _ {t, M _ {1}} ^ {\pi_ {1}} (s, a) \mid \mid p _ {t, M _ {2}} ^ {\pi_ {2}} (s, a)\right) \leq (t + 1) \epsilon_ {\pi} + \sum_ {t ^ {\prime} = 1} ^ {t} \mathbb {E} _ {s ^ {\prime}, a \sim p _ {t ^ {\prime} - 1, M _ {1}} ^ {\pi_ {1}} \left(s ^ {\prime}, a\right)} D _ {T V} \left(p _ {M _ {1}} \left(s \mid s ^ {\prime}, a\right) \mid \mid p _ {M _ {2}} \left(s \mid s ^ {\prime}, a\right)\right) \tag {25}
$$
Next, we move the result of Eq.(25) to Eq.(22), we can get:
$$
\begin{array}{l} V _ {M _ {2}} ^ {\pi_ {2}} - V _ {M _ {1}} ^ {\pi_ {1}} \\ \geq - 2 R _ {m a x} \sum_ {t = 0} ^ {\infty} \gamma^ {t} \left(\left(t + 1\right) \epsilon_ {\pi} + \sum_ {t ^ {\prime} = 1} ^ {t} \mathbb {E} _ {s ^ {\prime}, a \sim p _ {t ^ {\prime} - 1, M _ {1}} \left(s ^ {\prime}, a\right)} \right. D _ {T V} \left(p _ {M _ {1}} \left(s \mid s ^ {\prime}, a\right) \mid \mid p _ {M _ {2}} \left(s \mid s ^ {\prime}, a\right)\right) \tag {26} \\ = - 2 R _ {m a x} \left(\frac {\epsilon_ {\pi}}{(1 - \gamma) ^ {2}} + \frac {1}{1 - \gamma} \sum_ {t = 1} ^ {\infty} \gamma^ {t} \mathbb {E} _ {s ^ {\prime}, a \sim p _ {t - 1, M _ {1}} (s ^ {\prime}, a)} D _ {T V} \left(p _ {M _ {1}} (s | s ^ {\prime}, a) \mid \mid p _ {M _ {2}} (s | s ^ {\prime}, a)\right)\right) \\ \end{array}
$$
Here, we first simplify the second term of the Eq.(26)