SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\Lambda \left(s _ {s r c}, a _ {s r c}, s _ {s r c} ^ {\prime}\right) := \mathbb {P} \left(V _ {t a r} ^ {\pi} \left(s _ {s r c} ^ {\prime}\right) \mid \operatorname {M e a n} \left(\mathcal {Q} _ {t a r} ^ {\pi} \left(s _ {s r c}, a _ {s r c}\right)\right), \operatorname {V a r} \left(\mathcal {Q} _ {t a r} ^ {\pi} \left(s _ {s r c}, a _ {s r c}\right)\right)\right). \tag {6}
$$
Based on the likelihood estimation, we utilize the rejection sampling to select fixed percentage data (i.e.,  $25\%$ ) with the highest likelihood from a batch of source domain transitions at each training iteration. Specifically, we train the value function by optimizing the following objective:
$$
\theta \gets \underset {\theta} {\arg \min} \frac {1}{2} \mathbb {E} _ {(s, a, r, s ^ {\prime}) \sim D _ {t a r}} \left[ (Q _ {\theta} - \mathcal {T} Q _ {\theta}) ^ {2} \right] + \frac {1}{2} \mathbb {E} _ {(s, a, r, s ^ {\prime}) \sim D _ {s r c}} \left[ \omega (s, a, s ^ {\prime}) (Q _ {\theta} - \mathcal {T} Q _ {\theta}) ^ {2} \right],
$$
where  $\omega (s,a,s^{\prime}):= \mathbb{1}\left(\Lambda (s,a,s^{\prime}) > \Lambda_{\xi \%}\right).$  (7)
$\Lambda_{\xi \%}$  is the top  $\xi$ -quantile likelihood estimation of the minibatch sampled from source domain data,  $\mathcal{T}$  represents the Bellman operator, and  $D_{src}, D_{tar}$  denote replay buffers of two domains.
Consider the case when the agent can perform online interactions with the source domain, the training data mostly comes from the source domain, while we aim to train a policy for the target domain. Hence, exploring the source domain is essential to collect transitions that might be high-value concerning the target domain. Thus, we introduce an exploration policy  $\pi^{\mathrm{E}}$  that maximizes the approximate upper confidence bound of the  $Q$ -value, i.e.,  $\pi^{\mathrm{E}} \gets \arg \max_{\pi^{\mathrm{E}}} \mathbb{E}_{s \sim D_{tar} \cup D_{src}}[Q_{\mathrm{UB}}(s, a)|_{a \sim \pi^{\mathrm{E}}(\cdot|s)}]$ , where  $Q_{\mathrm{UB}}(s, a) \coloneqq \max \{Q_{\theta_i}(s, a)\}_{i=1}^2$  under the implementation with SAC [24] backbone. Importantly, the exploration policy  $\pi^{\mathrm{E}}$  is separate from the main policy  $\pi$  learned via vanilla SAC.  $\pi^{\mathrm{E}}$  and  $\pi$  are used for data collection in the source domain and target domain, respectively. The optimistic data collection technique has been proposed for advanced exploration [11] while we utilize the technique in online dynamics adaptation setting.
5.2 Adaptation with Offline Dataset of Source Domain
So far, we have discussed the setting where the agent can interact with the source domain to collect data actively. Nonetheless, simultaneous online access to the source and target domain might sometimes be impractical. In order to address the limitation, we aim to extend our method to the setting we refer to as Offline Source with Online Target, in which the agent can access a source domain offline dataset and a relatively small number of online interactions with the target domain.
To adapt VGDF to such a setting, we propose a novel theoretical result of the performance guarantee:
Theorem 5.1. Under the setting with offline source domain dataset  $D$  whose empirical estimation of the data collection policy is  $\pi_D(a|s) \coloneqq \frac{\sum_D \mathbb{1}(s, a)}{\sum_D \mathbb{1}(s)}$ , let  $\mathcal{M}_{src}$  and  $\mathcal{M}_{tar}$  denote the source and target domain, respectively. We have the performance guarantee of any policy  $\pi$  over the two MDPs:
$$
\eta_ {\mathcal {M} _ {t a r}} (\pi) \geq \eta_ {\mathcal {M} _ {s r c}} (\pi) - \frac {4 r _ {\operatorname* {m a x}}}{(1 - \gamma) ^ {2}} \underbrace {\mathbb {E} _ {\rho_ {\mathcal {M} _ {s r c}} , P _ {s r c}} [ D _ {T V} (\pi_ {D} | | \pi) ]} _ {(a): p o l i c y r e g u l a r i z a t i o n} - \frac {1}{1 - \gamma} \underbrace {\mathbb {E} _ {\rho_ {\mathcal {M} _ {s r c}}} [ \left| \zeta (s , a) \right| ]} _ {(b): v a l u e d i f f e r e n c e}, \tag {8}
$$
where  $\zeta(s, a) \coloneqq \mathbb{E}_{P_{src}, \pi}\left[Q_{\mathcal{M}_{tar}}^{\pi}(s', a')\right] - \mathbb{E}_{P_{tar}, \pi}\left[Q_{\mathcal{M}_{tar}}^{\pi}(s', a')\right]$ .
The proof of Theorem 5.1 is given in Appendix B. This theorem highlights the importance of policy regularization and value difference for achieving desirable performance. It is worth noting that the policy regularization term can shed light on the impact of behavior cloning, which has been proven effective for offline RL [22]. Additionally, the value difference term has a similar structure to that of Theorem 3. Thus, we propose a variant called  $VGDF + BC$  that combines behavior cloning loss with the original selective data sharing scheme. The pseudocode is shown in Algorithm 2, Appendix A.
In this section, we present empirical investigations of our approach. We examine the effectiveness of our method in scenarios with various dynamics shifts, including kinematic change and morphology change. Furthermore, we provide ablation studies and qualitative analysis of our method. Details of environment settings and the implementation are shown in Appendix D and Appendix E, respectively. Additional results are in Appendix F.
6.1 Adaptation Performance Evaluation
To systematically investigate the adaptation performance of the methods, we construct two types of dynamics shift scenarios, including kinematic shift and morphology shifts, for four environments (HalfCheetah, Ant, Walker, Hopper) from Gym Mujoco [71, 7]. We use the original environment as the source domain across all experiments. To simulate kinematic shifts, we limit the rotation angle range of specific joints to simulate the broken joint scenario. As for morphology shifts, we modify the size of specific limbs while the number of limbs keeps unchanged to ensure the state/action space consistent across domains. Full details of the environment settings are deferred to Appendix D.
We compare our algorithm with four baselines: (i) DARC [17] trains the domain classifiers to compensate the agent with an extra reward for seeking dynamics-consistent behaviors; (ii) GARAT [14] trains the policy with an adversarial imitation reward in the grounded source domain via action transformation [25]; (iii) IW Clip (Importance Weighting Clip) performs importance-weighted bellman updates for source domain samples. The importance weights (i.e.,  $P_{tar}(s'|s,a) / P_{src}(s'|s,a)$ ) are approximated by the domain classifiers proposed in DARC, and we clip the weight to  $[10^{-4}, 1]$  to stabilize training; (iv) Finetune uses the  $10^{5}$  target domain transitions to finetune the policy trained in the source domain with 1M samples. Furthermore, Zero-shot shows the performance of directly transferring the learned policy in the source domain to the target domain, and Oracle demonstrates the performance of the policy trained in the target domain from scratch with 1M transitions. We run all algorithms with the same five random seeds. The implementation details are given in Appendix E.1.
As the results in Figure 4 show, our method outperforms  $GARAT$  and  $IWClip$  in all environments.  $DARC$  demonstrates competitive performance only in the first two environments, while it does not work in other environments. We believe that the assumption of  $DARC$  does not hold in the failure cases due to the significant dynamics mismatch.  $GARAT$  fails in almost all environments, which we
Figure 4: Adaptation performance in the target domain with kinematic mismatch (Top) or morphology mismatch (Bottom). Solid curves are average returns over five runs with different random seeds, and shaded areas indicate one standard deviation. We use data ratio  $\Gamma = 10$ , which indicates all algorithms perform  $10^{6}$  online interactions with the source domain except Oracle.
believe is caused by the impractical action transformation from inaccurate dynamics models. The performance of Zero-shot suggests that the policies trained in the source domains barely work in the target domains due to dynamics mismatch. Finetune achieves promising results and outperforms our method in two of eight environments. We believe that the temporally-extended behaviors of the pre-trained policy benefit learning in the downstream tasks with the assistance of efficient exploration. Nonetheless, our method is the only one that outperforms or matches the asymptotic performance of Oracle in four out of eight environments.
To investigate the impact of design components in our method, we perform ablation analysis on the ratio of transitions  $\Gamma$ , data selection ratio  $\xi\%$ , and the optimistic exploration.
Data ratio  $\Gamma$ . We employ different ratios of transitions from the source domain versus those from the target domain ( $\Gamma = 5, 10, 20$ ) for variants of our algorithm. The results shown in Figure 5 demonstrate that the performance of our algorithm improves with more source domain transitions when the number of target domain transitions is the same. This finding indicates that VGDF can fully exploit the reusable
Figure 5: Effect of transition ratio  $\Gamma$ .
source domain transitions to enhance the training efficiency concerning the target domain.
Figure 6: Effect of data selection ratios  $\xi \%$
Figure 7: Effect of the optimistic exploration technique (i.e.,  $\pi^{\mathrm{E}}$ ).
Data selection ratio  $\xi\%$ . We employ different data ratios (10%, 25%, 50%, 75%) for the variants of our algorithm. Furthermore, we propose a baseline algorithm Mix that learns with all source domain samples without selection ( $\omega(s, a, s') \equiv 1$  in Eq. (7)). The results, shown in Figure 6, indicate
Table 1: Results in the offline source online target setting. We evaluate the algorithms via the performance of the learned policy in the target domain and report the mean and std of the results across five runs with different random seeds.
Offline only	Symmetric sampling	H2O	VGDF + BC
halfcheetah - broken back thigh	1128 ± 156	2439 ± 390	5761 ±148	4834 ± 250
halfcheetah - no thighs	361 ± 39	2211 ± 77	3023 ± 77	3910 ±160
hopper - broken hips	155 ± 19	2607 ± 181	2435 ± 325	2785 ±75
hopper - short feet	399 ± 5	2144 ± 509	868 ± 73	3060 ±60
walker - broken right thigh	1453 ± 412	709 ± 128	3743 ±50	3000 ± 388
walker - no right thigh	975 ± 131	872 ± 301	2600 ± 355	3293 ±306
that our algorithm performs robustly under various ratios within a specific range (e.g.,  $\xi \% \leq 50\%$ ). Surprisingly, Mix performs exceptionally well in environments with kinematic mismatches but fails in scenarios with morphology shifts. We attribute this to the less significant dynamics shift induced by kinematic changes compared to morphology changes.
Optimistic data collection. To validate the effect of the optimistic exploration  $\pi^{\mathrm{E}}$ , we introduce a variant of our method without  $\pi^{\mathrm{E}}$ . The results are shown in Figure 7. Removing the optimistic exploration technique results in performance degradation in three out of four environments concerning the sample efficiency, validating the effectiveness of the exploration policy.
6.3 Performance under Offline Source with Online Target
In this subsection, we extend our method to the setting with a source domain offline dataset and limited online interactions with the target domain, investigating the performance of our method without online access to the source domain. We use the D4RL medium datasets [20] of three environments (i.e., HalfCheetah, Walker, Hopper) for evaluation. We compare the proposed  $VGDF + BC$  (Section 5.2) with the following baselines: Offline only that directly transfers the offline learned policy via CQL [35] to the target domain; Symmetric sampling [3] that samples  $50\%$  of the data from the target domain replay buffer and the remaining  $50\%$  from the source domain offline dataset for each training step;  $H2O$  [49] that penalizes the Q function learning on source domain transitions with the estimated dynamics gap via learned classifiers. All algorithms have limited interactions with the target domain to  $10^5$  steps. The experimental details are shown in Appendix E.2. The results shown in Table 1 demonstrate that our method outperforms the other methods in four out of six environments, indicating that filtering the source domain data with the value consistency paradigm is effective in the offline-online setting.
6.4 Quantifying Dynamics Mismatch via Fictitious Value Proximity
Although the empirical results suggest that our method can adapt the policy in the face of various dynamics shifts, the degree of the dynamics mismatch can only be evaluated via the adaptation performance rather than be quantified directly. Here, we propose quantifying the dynamics shifts via the proposed Fictitious Value Proximity (FVP) (Section 5.1).
We approximate the FVP in Eq. (5) by calculating the average likelihood of a batch of samples from the source domain by  $\mathbb{E}[\Lambda(s, a, s')] \approx \frac{1}{B} \sum_{(s, a, s')} \hat{\Lambda}(s, a, s')$ . We show the approximated FVP in Ant environments with kinematic or morphology shifts in Figure 8. We observe a significant gap between the FVP values of the paired domains, which suggests the target domain with the morphology shifts is "closer" to the source domain than the target domain with the kinematic shifts with respect to the value difference. FVP measured by value differences quantifies the long-term effect on the expected return. Such a measurement can be regarded as a way to quantify the domain discrepancies.
Figure 8: Quantification analysis of the approximated FVP in Ant environments.
This work addresses the online dynamics adaptation problem by proposing VGDF that selectively shares the source domain transitions from a value consistency paradigm. Starting from the motivation example, we reveal the limitation of the prior dynamics-based method. Then we introduce a novel value discrepancy perspective with theoretical analysis, motivated by the insight that paired transitions with consistent value targets can be regarded as equivalent for training. Practically, we propose VGDF and the variant for the offline source domain setting. Empirical studies demonstrate the effectiveness of our method under significant dynamics gaps, including kinematics shifts and morphology shifts.
Limitation and future directions. One limitation of our method is the reliance on the ensemble dynamics models. However, the recent work estimating the epistemic uncertainty with a single model [19] could be applicable. Furthermore, value-aware model learning [18] may improve our method by training dynamics models with accurate value predictions of the generated samples. Exploring the effectiveness of value consistency for generalizing across reward functions can be another direction for future research. Finally, validating the effectiveness of the data sharing method in the Sim2Real problem would contribute to the robotics community. The online interaction with the reality system could be risky, recent works [6, 70] can be integrated for safe online interactions.
This work is supported by the National Natural Science Foundation of China (Grant No.62306242), the National Key R&D Program of China (Grant No.2022ZD0160100), Shanghai Artificial Intelligence Laboratory, Shanghai Municipal Science and Technology Major Project (No.2021SHZDZX0103), Scientific Research Development Center in Higher Education Institutions by the Ministry of Education, China (No.2021ITA10013), Shanghai Engineering Research Center of AI and Robotics, Engineering Research Center of AI and Robotics, Ministry of Education, China. We would like to thank the anonymous reviewers for their valuable suggestions.
[1] Ilge Akkaya, Marcin Andrychowicz, Maciek Chogiej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik's cube with a robot hand. arXiv preprint arXiv:1910.07113, 2019.
[2] Chenjia Bai, Lingxiao Wang, Lei Han, Jianye Hao, Animesh Garg, Peng Liu, and Zhaoran Wang. Principled exploration via optimistic bootstrapping and backward induction. In International Conference on Machine Learning, pages 577-587. PMLR, 2021.
[3] Philip J Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data. arXiv preprint arXiv:2302.02948, 2023.
[4] Mohammadamin Barekatain, Ryo Yonetani, and Masashi Hamaya. Multipolar: multi-source policy aggregation for transfer reinforcement learning between diverse environmental dynamics. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pages 3108-3116, 2021.
[5] Andre Barreto, Diana Borsa, John Quan, Tom Schaul, David Silver, Matteo Hessel, Daniel Mankowitz, Augustin Zidek, and Remi Munos. Transfer in deep reinforcement learning using successor features and generalised policy improvement. In International Conference on Machine Learning, pages 501-510. PMLR, 2018.
[6] Homanga Bharadhwaj, Aviral Kumar, Nicholas Rhinehart, Sergey Levine, Florian Shkurti, and Animesh Garg. Conservative safety critics for exploration. In International Conference on Learning Representations, 2020.
[7] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016.
[8] Yevgen Chebotar, Ankur Handa, Viktor Makoviychuk, Miles Macklin, Jan Issac, Nathan Ratliff, and Dieter Fox. Closing the sim-to-real loop: Adapting simulation randomization with real
A Algorithm Description
The pseudocode of  $VGDF$  is presented in Algorithm 1. We utilize SAC [24] as our backbone algorithm. We employ a fixed entropy temperature coefficient in all experiments, demonstrating sufficient empirical performance. The training of the dynamics model ensemble follows prior works [10, 30] with the MLE loss. The calculation of the Fictitious Value Proximity follows Eq. (6) proposed in Section 5.1. Furthermore, the pseudocode of  $VGDF + BC$  is presented in Algorithm 2. We introduce the value-normalized tradeoff between the behavior cloning loss and the policy gradient following the prior work [22].
Algorithm 1 Value-Guided Data Filtering (VGDF)
Input: Source domain  $\mathcal{M}_{src}$ , target domain  $\mathcal{M}_{tar}$ , and transition ratio  $\Gamma (= 10)$  (source vs. target). Initialization: Policy  $\pi$ , exploration policy  $\pi^{\mathrm{E}}$ , value functions  $\{Q_{\theta_i}\}_{i=1,2}$ , replay buffers  $\{D_{src}, D_{tar}\}$ , dynamics model ensemble  $\{T_{\phi_i}\}_{i=1}^{M}$ , data selection ratio  $\xi$ , batch size  $B$ , entropy temperature coefficient  $\lambda$ .
1: for  $t = 1, 2, \ldots$  do
2: #Interact with the source domain
3: Sample transition  $(s_{src}, a_{src}, r_{src}, s_{src}^{\prime})$  using  $\pi^{\mathrm{E}}$  in  $\mathcal{M}_{src}$
4:  $D_{src} \gets D_{src} \cup (s_{src}, a_{src}, r_{src}, s_{src}')$
5: #Interact with the target domain
6: if  $t\% \Gamma == 0$  then
7: Sample transition  $(s_{tar}, a_{tar}, r_{tar}, s_{tar}')$  using  $\pi$  in  $\mathcal{M}_{tar}$
8:  $D_{tar} \gets D_{tar} \cup (s_{tar}, a_{tar}, r_{tar}, s_{tar}')$
9: end if
10: Optimize dynamics ensemble  $\{T_{\phi_i}\}_{i = 1}^M$  with  $D_{tar}$  via Eq. (13)
11: Sample  $b_{src} \coloneqq \{(s, a, r, s')\}_{src}^{B}$  from  $D_{src}$
12: Sample  $b_{tar} \coloneqq \{(s, a, r, s')\}_{tar}^{B}$  from  $D_{tar}$
13: Obtain Fictitious Value Proximity (FVP)  $\{\Lambda(s, a, s')\}^B$  via Eq. (6) for transitions in  $b_{src}$ .
14: Obtain FVP quantile  $\Lambda_{\xi \%}$  of  $\{\Lambda (s,a,s^{\prime})\}^{B}$
15: #Optimize value function with data filtering
16:  $\theta_{i = 1,2}\gets \underset {\theta_i}{\arg \min}\frac{1}{2B}\sum_{b_{tar}}\left[(Q_{\theta_i} - \mathcal{T}Q_{\theta_i})^2\right] +$
17:  $\frac{1}{[2B\cdot\xi\%]} \sum_{b_{src}} \left[\mathbb{1}\left(\Lambda(s, a, s') > \Lambda_{\xi\%}\right) (Q_{\theta_i} - \mathcal{T}Q_{\theta_i})^2\right]$
18: #Optimize policies
19:  $\pi^{\mathrm{E}}\gets \arg \max_{\pi^{\mathrm{E}}}\frac{1}{2B}\sum_{b_{tar}\cup b_{src}}\Big[\max \left\{Q_{\theta_1}(s,a),Q_{\theta_2}(s,a)\right\} |_{a\sim \pi^{\mathrm{E}}(\cdot |\cdot s)} + \lambda \mathcal{H}[\pi^{\mathrm{E}}]\Big]$
20:  $\pi \gets \arg \max_{\pi} \frac{1}{2B} \sum_{b_{tar} \cup b_{src}} \left[ \min \left\{Q_{\theta_1}(s, a), Q_{\theta_2}(s, a)\right\} \mid_{a \sim \pi(\cdot|s)} + \lambda \mathcal{H}[\pi] \right]$
21: end for
B Proofs of the Performance Guarantees
This section presents the proof of our main results. Specifically, we propose that the value discrepancy can be leveraged for the performance guarantee across different domains Lemma C.3. In Theorem B.1, we convert the performance bound induced by the value discrepancy into a novel form for the offline source domain setting.
Theorem B.1. (Performance bound controlled by dynamics discrepancy.) Denote the source domain and target domain with different dynamics as  $\mathcal{M}_{src}$  and  $\mathcal{M}_{tar}$ , respectively. We have the
Algorithm 2 Value-Guided Data Filtering + Behavior Cloning (VGDF + BC)
Input: Source domain offline dataset  $D_{src}$ , target domain  $\mathcal{M}_{tar}$ , max interaction steps with the target domain  $T_{\mathrm{max}}$ , and transition ratio  $\Gamma$  ( $:= \frac{|D_{src}|}{T_{\mathrm{max}}} = 10$ ) (source vs. target).
Initialization: Policy  $\pi$ , value functions  $\{Q_{\theta_i}\}_{i=1,2}$ , target domain replay buffer  $D_{tar}$ , dynamics model ensemble  $\{T_{\phi_i}\}_{i=1}^M$ , data selection ratio  $\xi$ , batch size  $B$ , entropy temperature coefficient  $\lambda$ , train repeat  $K$ , behavior cloning constant  $\alpha$ .
1: for  $t = 1,2,\ldots ,T_{\max}$  do
2: #Interact with the target domain
3: Sample transition  $(s_{tar}, a_{tar}, r_{tar}, s_{tar}')$  using  $\pi$  in  $\mathcal{M}_{tar}$
4:  $D_{tar} \gets D_{tar} \cup (s_{tar}, a_{tar}, r_{tar}, s_{tar}')$
5: #Repeat training for  $K$  times per step
6: for  $k = 1, 2, \ldots, K$  do
7: Optimize dynamics ensemble  $\{T_{\phi_i}\}_{i = 1}^M$  with  $D_{tar}$  via Eq. (13)
8: Sample  $b_{src} \coloneqq \{(s, a, r, s')\}_{src}^{B}$  from  $D_{src}$
9: Sample  $b_{tar} \coloneqq \{(s, a, r, s')\}_{tar}^{B}$  from  $D_{tar}$
10: Obtain Fictitious Value Proximity (FVP)  $\{\Lambda(s, a, s')\}^B$  via Eq. (6) for transitions in  $b_{src}$ .
11: Obtain FVP quantile  $\Lambda_{\xi \%}$  of  $\{\Lambda (s,a,s^{\prime})\} ^{B}$
12: #Optimize value function with data filtering
13:  $\theta_{i = 1,2}\gets \underset {\theta_i}{\arg \min}\frac{1}{2B}\sum_{b_{tar}}\left[(Q_{\theta_i} - \mathcal{T}Q_{\theta_i})^2\right] +$
14:  $\frac{1}{[2B\cdot\xi\%]} \sum_{b_{src}} \left[\mathbb{1}\left(\Lambda(s, a, s') > \Lambda_{\xi\%}\right) (Q_{\theta_i} - \mathcal{T}Q_{\theta_i})^2\right]$
15: #Optimize policy with behavior cloning regularization
16:  $\beta = \alpha / \left\{\frac{1}{2B}\sum_{b_{tar} \cup b_{src}}\left[\left|\min\left\{Q_{\theta_1}(s,a), Q_{\theta_2}(s,a)\right\}_{a \sim \pi(\cdot|s)}\right|\right]\right\}$
17:  $\pi \gets \arg \max_{\pi} \frac{\beta}{2B} \sum_{b_{tar} \cup b_{src}} \left[ \min \{Q_{\theta_1}(s, a), Q_{\theta_2}(s, a) \}_{a \sim \pi(\cdot|s)} + \lambda \mathcal{H}[\pi] \right] -$
18:  $\frac{1}{B}\sum_{(s,a)\sim b_{src}}\left[(\pi (s) - a)^2\right]$
19: end for
performance difference of any policy  $\pi$  evaluated under  $\mathcal{M}_{src}$  and  $\mathcal{M}_{tar}$  be bounded as below,
$$
\eta_ {\mathcal {M} _ {t a r}} (\pi) \geq \eta_ {\mathcal {M} _ {s r c}} (\pi) - \frac {2 \gamma r _ {\operatorname* {m a x}}}{(1 - \gamma) ^ {2}} \cdot \mathbb {E} _ {\rho_ {s r c} ^ {\pi}} \left[ D _ {\mathrm {T V}} \left(P _ {s r c} (\cdot | s, a) \| P _ {t a r} (\cdot | s, a)\right) \right].
$$