SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \eta_ {s r c} (\pi) - \eta_ {t a r} (\pi) = \frac {\gamma}{1 - \gamma} \mathbb {E} _ {\rho_ {s r c} ^ {\pi} (s, a)} \left[ \int_ {s ^ {\prime}} P _ {s r c} \left(s ^ {\prime} \mid s, a\right) V _ {t a r} ^ {\pi} \left(s ^ {\prime}\right) - \int_ {s ^ {\prime}} P _ {t a r} \left(s ^ {\prime} \mid s, a\right) V _ {t a r} ^ {\pi} \left(s ^ {\prime}\right) d s ^ {\prime} \right] (L e m m a \\ = \frac {\gamma}{1 - \gamma} \mathbb {E} _ {\rho_ {s r c} ^ {\pi} (s, a)} \left[ \int_ {s ^ {\prime}} \left(P _ {s r c} \left(s ^ {\prime} | s, a\right) - P _ {t a r} \left(s ^ {\prime} | s, a\right)\right) V _ {t a r} ^ {\pi} \left(s ^ {\prime}\right) d s ^ {\prime} \right] \\ \leq \frac {\gamma}{1 - \gamma} \mathbb {E} _ {\rho_ {s r c} ^ {\pi} (s, a)} \left[ \int_ {s ^ {\prime}} | \left(P _ {s r c} \left(s ^ {\prime} \mid s, a\right) - P _ {t a r} \left(s ^ {\prime} \mid s, a\right)\right) V _ {t a r} ^ {\pi} \left(s ^ {\prime}\right) | d s ^ {\prime} \right] \\ \leq \frac {\gamma}{1 - \gamma} \cdot \frac {r _ {\operatorname* {m a x}}}{1 - \gamma} \mathbb {E} _ {\rho_ {s r c} ^ {\pi} (s, a)} \left[ \int_ {s ^ {\prime}} | P _ {s r c} (s ^ {\prime} | s, a) - P _ {t a r} (s ^ {\prime} | s, a) | d s ^ {\prime} \right] \\ = \frac {2 \gamma r _ {\operatorname* {m a x}}}{(1 - \gamma) ^ {2}} \mathbb {E} _ {\rho_ {s r c} ^ {\pi} (s, a)} \left[ D _ {\mathrm {T V}} \left(P _ {s r c} (\cdot | s, a) \| P _ {t a r} (\cdot | s, a)\right) \right]. \tag {9} \\ \end{array}
$$
Theorem B.2. (Performance bound controlled by value difference.) Denote the source domain and target domain as  $\mathcal{M}_{src}$  and  $\mathcal{M}_{tar}$ , respectively. We have the performance guarantee of any policy  $\pi$  over the two MDPs:
$$
\eta_ {\mathcal {M} _ {t a r}} (\pi) \geq \eta_ {\mathcal {M} _ {s r c}} (\pi) - \frac {\gamma}{1 - \gamma} \cdot \mathbb {E} _ {\rho_ {\mathcal {M} _ {s r c}} ^ {\pi}} \left[ \left| \mathbb {E} _ {P _ {s r c}} \left[ V _ {\mathcal {M} _ {t a r}} ^ {\pi} (s ^ {\prime}) \right] - \mathbb {E} _ {P _ {t a r}} \left[ V _ {\mathcal {M} _ {t a r}} ^ {\pi} (s ^ {\prime}) \right] \right| \right].
$$
$$
\begin{array}{l} \eta_ {s r c} (\pi) - \eta_ {t a r} (\pi) = \frac {\gamma}{1 - \gamma} \mathbb {E} _ {\rho_ {s r c} ^ {\pi} (s, a)} \left[ \int_ {s ^ {\prime}} P _ {s r c} (s ^ {\prime} | s, a) V _ {\mathcal {M} _ {t a r}} ^ {\pi} (s ^ {\prime}) - \int_ {s ^ {\prime}} P _ {t a r} (s ^ {\prime} | s, a) V _ {\mathcal {M} _ {t a r}} ^ {\pi} (s ^ {\prime}) d s ^ {\prime} \right] (L e m m a \\ = \frac {\gamma}{1 - \gamma} \cdot \mathbb {E} _ {\rho_ {\mathcal {M} _ {s r c}} ^ {\pi}} \left[ \mathbb {E} _ {P _ {s r c}} \left[ V _ {\mathcal {M} _ {t a r}} ^ {\pi} (s ^ {\prime}) \right] - \mathbb {E} _ {P _ {t a r}} \left[ V _ {\mathcal {M} _ {t a r}} ^ {\pi} (s ^ {\prime}) \right] \right] \\ \leq \frac {\gamma}{1 - \gamma} \cdot \mathbb {E} _ {\rho_ {\mathcal {M} _ {s r c}} ^ {\pi}} \left[ \left| \mathbb {E} _ {P _ {s r c}} \left[ V _ {\mathcal {M} _ {t a r}} ^ {\pi} (s ^ {\prime}) \right] - \mathbb {E} _ {P _ {t a r}} \left[ V _ {\mathcal {M} _ {t a r}} ^ {\pi} (s ^ {\prime}) \right] \right| \right] \\ \end{array}
$$
Theorem B.3. Under the setting with offline source domain dataset  $D$  whose empirical estimation of the data collection policy is  $\pi_D(a|s) \coloneqq \frac{\sum_D \mathbb{1}(s, a)}{\sum_D \mathbb{1}(s)}$ , let  $\mathcal{M}_{src}$  and  $\mathcal{M}_{tar}$  denote the source and target domain, respectively. We have the performance guarantee of any policy  $\pi$  over the two MDPs:
$$
\eta_ {\mathcal {M} _ {t a r}} (\pi) \geq \eta_ {\mathcal {M} _ {s r c}} (\pi) - \frac {4 r _ {\operatorname* {m a x}}}{(1 - \gamma) ^ {2}} \mathbb {E} _ {\rho_ {\mathcal {M} _ {s r c}}, P _ {s r c}} \left[ D _ {T V} \left(\pi_ {D} \mid \mid \pi\right) \right] - \frac {1}{1 - \gamma} \mathbb {E} _ {\rho_ {\mathcal {M} _ {s r c}}} \left[ \left| \zeta (s, a) \right| \right], \tag {10}
$$
$$
\eta_ {\mathcal {M} _ {t a r}} (\pi) - \eta_ {\mathcal {M} _ {s r c}} (\pi) = \underbrace {\left(\eta_ {\mathcal {M} _ {s r c}} (\pi_ {D}) - \eta_ {\mathcal {M} _ {s r c}} (\pi)\right)} _ {(a)} - \underbrace {\left(\eta_ {\mathcal {M} _ {s r c}} (\pi_ {D}) - \eta_ {\mathcal {M} _ {t a r}} (\pi)\right)} _ {(b)}.
$$
$$
\begin{array}{l} \eta_{\mathcal{M}_{src}}(\pi_{D}) - \eta_{\mathcal{M}_{src}}(\pi)\geq -\frac{1}{1 - \gamma}\mathbb{E}_{\substack{s,a\sim \rho_{\mathcal{M}_{src}}^{\pi_{D}}\\ s^{\prime}\sim P_{src}(\cdot |s,a)}}\left[  \left|\mathbb{E}_{a^{\prime}\sim \pi_{D}(\cdot |s^{\prime})}\left[Q_{\mathcal{M}_{src}}^{\pi}(s^{\prime},a^{\prime})\right] - \mathbb{E}_{a^{\prime}\sim \pi (\cdot |s^{\prime})}\left[Q_{\mathcal{M}_{src}}^{\pi}(s^{\prime},a^{\prime})\right]\right| \right] \\ = -\frac{1}{1 - \gamma}\mathbb{E}_{\substack{s,a\sim \rho_{\mathcal{M}_{src}}^{\pi_{D}}\\ s^{\prime}\sim P_{src}(\cdot |s,a)}}\left[\Bigg|\sum_{\mathcal{A}}\left(\pi_{D}(a^{\prime}|s^{\prime}) - \pi (a^{\prime}|s^{\prime})\right)Q_{\mathcal{M}_{src}}^{\pi}(s^{\prime},a^{\prime})\Bigg|\right] \\ \geq -\frac{1}{1 - \gamma}\mathbb{E}_{\substack{s,a\sim \rho_{\mathcal{M}_{src}}^{\pi_{D}}\\ s^{\prime}\sim P_{src}(\cdot |s,a)}}\left[\Bigg|\sum_{\mathcal{A}}\big(\pi_{D}(a^{\prime}|s^{\prime}) - \pi (a^{\prime}|s^{\prime})\big)\frac{r_{\max}}{1 - \gamma}\Bigg|\right] \\ \geq -\frac{r_{\max}}{(1 - \gamma)^{2}}\mathbb{E}_{\substack{s,a\sim \rho^{\pi_{D}}_{\mathcal{M}_{src}\\ s^{\prime}\sim P_{src}(\cdot |s,a)}}}\left[\sum_{\mathcal{A}}|\pi_{D}(a^{\prime}|s^{\prime}) - \pi (a^{\prime}|s^{\prime})|\right] \\ = - \frac {2 r _ {\mathrm {m a x}}}{(1 - \gamma) ^ {2}} \mathbb {E} _ {s, a \sim \rho_ {\mathcal {M} _ {s r c}} ^ {\pi_ {D}}} \left[ D _ {T V} \left(\pi_ {D} (\cdot | s ^ {\prime}) \parallel \pi (\cdot | s ^ {\prime})\right) \right], \\ \end{array}
$$
$$
\begin{array}{l} \left. - \left(\eta_ {\mathcal {M} _ {s r c}} (\pi_ {D}) - \eta_ {\mathcal {M} _ {t a r}} (\pi)\right) \right. \\ = - \frac {1}{1 - \gamma} \mathbb {E} _ {s, a \sim \rho_ {\mathcal {M} _ {s r c}} ^ {\pi_ {D}}} \left[ \mathcal {G} _ {\mathcal {M} _ {1}, \mathcal {M} _ {2}} ^ {\pi_ {1}, \pi_ {2}} (s, a) \right] \tag {Lemma C.2} \\ \geq -\frac{2r_{max}}{(1 - \gamma)^{2}}\mathbb{E}_{\substack{s,a\sim \rho_{\mathcal{M}_{src}}^{\pi_{D}}\\ s^{\prime}\sim P_{src}(\cdot |s,a)}}[D_{TV}(\pi_{D}(\cdot |s^{\prime})  \|   \pi (\cdot |s^{\prime}))] \\ - \frac {1}{1 - \gamma} \mathbb {E} _ {s, a \sim \rho_ {\mathcal {M} _ {s r c}} ^ {\pi_ {D}}} \left[ \left| \mathbb {E} _ {s ^ {\prime}, a ^ {\prime} \sim P _ {s r c}, \pi} \left[ Q _ {\mathcal {M} _ {t a r}} ^ {\pi} \left(s ^ {\prime}, a ^ {\prime}\right) \right] - \mathbb {E} _ {s ^ {\prime}, a ^ {\prime} \sim P _ {t a r}, \pi} \left[ Q _ {\mathcal {M} _ {t a r}} ^ {\pi} \left(s ^ {\prime}, a ^ {\prime}\right) \right] \right| \right]. (L e m m a C. 3) \\ \end{array}
$$
Combining the two inequalities above completes the proof.
C Proofs of Lemmas
This section provides proof of several lemmas used for our theoretical results. The first lemma is adopted from [44], and the proof is essentially the same as the original paper. Lemma C.2 and Lemma C.3 support the derivation of the performance difference bound in Theorem B.3.
Lemma C.1. (Telescoping Lemma, Lemma 4.3 in [44].) Let  $\mathcal{M}_1 \coloneqq (\mathcal{S}, \mathcal{A}, P_1, r, \gamma)$  and  $\mathcal{M}_2 \coloneqq (\mathcal{S}, \mathcal{A}, P_2, r, \gamma)$  be two MDPs with different dynamics  $P_1$  and  $P_2$ . Given a policy  $\pi$ , let
$$
\mathcal {G} _ {\mathcal {M} _ {1}, \mathcal {M} _ {2}} ^ {\pi} (s, a) := \mathbb {E} _ {s ^ {\prime} \sim P _ {1}} \left[ V _ {\mathcal {M} _ {2}} ^ {\pi} (s ^ {\prime}) \right] - \mathbb {E} _ {s ^ {\prime} \sim P _ {2}} \left[ V _ {\mathcal {M} _ {2}} ^ {\pi} (s ^ {\prime}) \right],
$$
$$
\eta_ {\mathcal {M} _ {1}} (\pi) - \eta_ {\mathcal {M} _ {2}} (\pi) = \frac {\gamma}{(1 - \gamma)} \mathbb {E} _ {s, a \sim \rho_ {\mathcal {M} _ {1}} ^ {\pi}} \left[ \mathcal {G} _ {\mathcal {M} _ {1}, \mathcal {M} _ {2}} ^ {\pi} (s, a) \right].
$$
Proof. Define  $W_{j}$  as the expected return when executing  $\pi$  on  $\mathcal{M}_1$  for the first  $j$  steps, then switching to  $\pi$  and  $\mathcal{M}_2$  for the remainder. That is
$$
W_{j}:= \sum_{t = 0}^{\infty}\gamma^{t}\mathbb{E}_{\substack{t <   j:s_{t},a_{t}\sim P_{1},\pi \\ t\geq j:s_{t},a_{t}\sim P_{2},\pi_{2}}}[r(s_{t},a_{t})] = \mathbb{E}_{\substack{t <   j:s_{t},a_{t}\sim P_{1},\pi \\ t\geq j:s_{t},a_{t}\sim P_{2},\pi}}\left[\sum_{t = 0}^{\infty}\gamma^{t}r(s_{t},a_{t})\right].
$$
$$
W _ {0} = \mathbb {E} _ {s, a \sim \rho_ {\mathcal {M} _ {2}, \pi}} [ r (s _ {t}, a _ {t}) ] = \eta_ {\mathcal {M} _ {2}} (\pi),
$$
$$
\text {a n d} W _ {\infty} = \mathbb {E} _ {s, a \sim \rho_ {\mathcal {M} _ {1}, \pi}} \left[ r \left(s _ {t}, a _ {t}\right) \right] = \eta_ {\mathcal {M} _ {1}} (\pi).
$$
$$
\eta_ {\mathcal {M} _ {1}} (\pi) - \eta_ {\mathcal {M} _ {2}} (\pi) = \sum_ {j = 0} ^ {\infty} \left(W _ {j + 1} - W _ {j}\right). \tag {11}
$$
Convert  $W_{j}$  and  $W_{j + 1}$  as following:
$$
W _ {j} = R _ {j} + \mathbb {E} _ {s _ {j}, a _ {j} \sim P _ {1, \pi}} \left[ \mathbb {E} _ {s _ {j + 1} \sim P _ {2}} \left[ \gamma^ {j + 1} V _ {\mathcal {M} _ {2}} ^ {\pi} (s _ {j + 1}) \right] \right]
$$
$$
W _ {j + 1} = R _ {j} + \mathbb {E} _ {s _ {j}, a _ {j} \sim P _ {1}, \pi} \left[ \mathbb {E} _ {s _ {j + 1} \sim P _ {1}} \left[ \gamma^ {j + 1} V _ {\mathcal {M} _ {2}} ^ {\pi} (s _ {j + 1}) \right] \right]
$$
Plug back to Eq.11 and we obtain
$$
\begin{array}{l} \eta_ {\mathcal {M} _ {1}} (\pi) - \eta_ {\mathcal {M} _ {2}} (\pi) = \sum_ {j = 0} ^ {\infty} \left(W _ {j + 1} - W _ {j}\right) \\ = \sum_ {j = 0} ^ {\infty} \gamma^ {j + 1} \mathbb {E} _ {s, a \sim \mathbb {P} _ {\mathcal {M} _ {1}, j} ^ {\pi}} \left[ \mathbb {E} _ {s ^ {\prime} \sim P _ {1}} \left[ V _ {\mathcal {M} _ {2}} ^ {\pi} (s ^ {\prime}) \right] - \mathbb {E} _ {s ^ {\prime} \sim P _ {2}} \left[ V _ {\mathcal {M} _ {2}} ^ {\pi} (s ^ {\prime}) \right] \right] \\ = \frac {\gamma}{(1 - \gamma)} \mathbb {E} _ {s, a \sim \rho_ {\mathcal {M} _ {1}} ^ {\pi}} \left[ \mathbb {E} _ {s ^ {\prime} \sim P _ {1}} \left[ V _ {\mathcal {M} _ {2}} ^ {\pi} (s ^ {\prime}) \right] - \mathbb {E} _ {s ^ {\prime} \sim P _ {2}} \left[ V _ {\mathcal {M} _ {2}} ^ {\pi} (s ^ {\prime}) \right] \right] \\ = \frac {\gamma}{(1 - \gamma)} \mathbb {E} _ {s, a \sim \rho_ {\mathcal {M} _ {1}} ^ {\pi}} \left[ \mathcal {G} _ {\mathcal {M} _ {1}, \mathcal {M} _ {2}} ^ {\pi} (s, a) \right]. \\ \end{array}
$$
Lemma C.2. (Extension of Telescoping Lemma.) Let  $\mathcal{M}_1 \coloneqq (\mathcal{S}, \mathcal{A}, P_1, r, \gamma)$  and  $\mathcal{M}_2 \coloneqq (\mathcal{S}, \mathcal{A}, P_2, r, \gamma)$  be two MDPs with different dynamics  $P_1$  and  $P_2$ . Given two policies  $\pi_1$ ,  $\pi_2$ , let
$$
\mathcal {G} _ {\mathcal {M} _ {1}, \mathcal {M} _ {2}} ^ {\pi_ {1}, \pi_ {2}} (s, a) := \mathbb {E} _ {s ^ {\prime}, a ^ {\prime} \sim P _ {1}, \pi_ {1}} \left[ Q _ {\mathcal {M} _ {2}} ^ {\pi_ {2}} (s ^ {\prime}, a ^ {\prime}) \right] - \mathbb {E} _ {s ^ {\prime}, a ^ {\prime} \sim P _ {2}, \pi_ {2}} \left[ Q _ {\mathcal {M} _ {2}} ^ {\pi_ {2}} (s ^ {\prime}, a ^ {\prime}) \right],
$$
$$
\eta_ {\mathcal {M} _ {1}} (\pi_ {1}) - \eta_ {\mathcal {M} _ {2}} (\pi_ {2}) = \frac {1}{(1 - \gamma)} \mathbb {E} _ {s, a \sim \rho_ {\mathcal {M} _ {1}} ^ {\pi_ {1}}} \left[ \mathcal {G} _ {\mathcal {M} _ {1}, \mathcal {M} _ {2}} ^ {\pi_ {1}, \pi_ {2}} (s, a) \right].
$$
Proof. Define  $W_{j}$  as the expected return when executing  $\pi_{1}$  on  $\mathcal{M}_1$  for the first  $j$  steps, then switching to  $\pi_{2}$  and  $\mathcal{M}_2$  for the remainder. That is
$$
W _ {j} := \sum_ {t = 0} ^ {\infty} \gamma^ {t} \mathbb {E} _ {t \geq j: s _ {t}, a _ {t} \sim P _ {1}, \pi_ {1}} [ r (s _ {t}, a _ {t}) ] = \mathbb {E} _ {t \geq j: s _ {t}, a _ {t} \sim P _ {1}, \pi_ {1}} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} r (s _ {t}, a _ {t}) \right].
$$
$$
W _ {0} = \mathbb {E} _ {s, a \sim \rho_ {\mathcal {M} _ {2}, \pi_ {2}}} \left[ r (s _ {t}, a _ {t}) \right] = \eta_ {\mathcal {M} _ {2}} (\pi_ {2}),
$$
$$
\text {a n d} W _ {\infty} = \mathbb {E} _ {s, a \sim \rho_ {\mathcal {M} _ {1}, \pi_ {1}}} [ r (s _ {t}, a _ {t}) ] = \eta_ {\mathcal {M} _ {2}} (\pi_ {1}).
$$
$$
\eta_ {\mathcal {M} _ {1}} \left(\pi_ {1}\right) - \eta_ {\mathcal {M} _ {2}} \left(\pi_ {2}\right) = \sum_ {j = 0} ^ {\infty} \left(W _ {j + 1} - W _ {j}\right). \tag {12}
$$
$$
W _ {j} = R _ {j} + \mathbb {E} _ {s _ {j}, a _ {j} \sim P _ {1}, \pi_ {1}} \left[ \mathbb {E} _ {s _ {j + 1}, a _ {j + 1} \sim P _ {2}, \pi_ {2}} \left[ \gamma^ {j + 1} Q _ {\mathcal {M} _ {2}} ^ {\pi_ {2}} (s _ {j + 1}, a _ {j + 1}) \right] \right]
$$
$$
W _ {j + 1} = R _ {j} + \mathbb {E} _ {s _ {j}, a _ {j} \sim P _ {1}, \pi_ {1}} \left[ \mathbb {E} _ {s _ {j + 1}, a _ {j + 1} \sim P _ {1}, \pi_ {1}} \left[ \gamma^ {j + 1} Q _ {\mathcal {M} _ {2}} ^ {\pi_ {2}} (s _ {j + 1}, a _ {j + 1}) \right] \right]
$$
Plug back to Eq.12 and we obtain
$$
\begin{array}{l} \eta_ {\mathcal {M} _ {1}} (\pi_ {1}) - \eta_ {\mathcal {M} _ {2}} (\pi_ {2}) = \sum_ {j = 0} ^ {\infty} (W _ {j + 1} - W _ {j}) \\ = \sum_ {j = 0} ^ {\infty} \gamma^ {j + 1} \mathbb {E} _ {s, a \sim \mathbb {P} _ {\mathcal {M} _ {1}, j} ^ {\pi_ {1}}} \left[ \mathbb {E} _ {s ^ {\prime}, a ^ {\prime} \sim P _ {1}, \pi_ {1}} \left[ Q _ {\mathcal {M} _ {2}} ^ {\pi_ {2}} \left(s ^ {\prime}, a ^ {\prime}\right) \right] - \mathbb {E} _ {s ^ {\prime}, a ^ {\prime} \sim P _ {2}, \pi_ {2}} \left[ Q _ {\mathcal {M} _ {2}} ^ {\pi_ {2}} \left(s ^ {\prime}, a ^ {\prime}\right) \right] \right] \\ = \frac {\gamma}{(1 - \gamma)} \mathbb {E} _ {s, a \sim \rho_ {\mathcal {M} _ {1}} ^ {\pi_ {1}}} \left[ \mathbb {E} _ {s ^ {\prime}, a ^ {\prime} \sim P _ {1}, \pi_ {1}} \left[ Q _ {\mathcal {M} _ {2}} ^ {\pi_ {2}} (s ^ {\prime}, a ^ {\prime}) \right] - \mathbb {E} _ {s ^ {\prime}, a ^ {\prime} \sim P _ {2}, \pi_ {2}} \left[ Q _ {\mathcal {M} _ {2}} ^ {\pi_ {2}} (s ^ {\prime}, a ^ {\prime}) \right] \right] \\ = \frac {\gamma}{(1 - \gamma)} \mathbb {E} _ {s, a \sim \rho_ {\mathcal {M} _ {1}} ^ {\pi_ {1}}} \left[ \mathcal {G} _ {\mathcal {M} _ {1}, \mathcal {M} _ {2}} ^ {\pi_ {1}, \pi_ {2}} (s, a) \right]. \\ \end{array}
$$
Lemma C.3. (Bound of  $\mathcal{G}_{\mathcal{M}_1,\mathcal{M}_2}^{\pi_1,\pi_2}(s,a).$  ) Let
$$
\begin{array}{l} \mathcal {G} _ {\mathcal {M} _ {1}, \mathcal {M} _ {2}} ^ {\pi_ {1}, \pi_ {2}} (s, a) \leq \frac {2 r _ {\max }}{1 - \gamma} \mathbb {E} _ {s ^ {\prime} \sim P _ {1}} \left[ D _ {T V} \left(\pi_ {1} (\cdot | s ^ {\prime}) \| \pi_ {2} (\cdot | s ^ {\prime})\right) \right] \\ + \left| \mathbb {E} _ {s ^ {\prime}, a ^ {\prime} \sim P _ {1}, \pi_ {2}} \left[ Q _ {\mathcal {M} _ {2}} ^ {\pi_ {2}} (s ^ {\prime}, a ^ {\prime}) \right] - \mathbb {E} _ {s ^ {\prime}, a ^ {\prime} \sim P _ {2}, \pi_ {2}} \left[ Q _ {\mathcal {M} _ {2}} ^ {\pi_ {2}} (s ^ {\prime}, a ^ {\prime}) \right] \right|. \\ \end{array}
$$
$$
\begin{array}{l} \mathcal {G} _ {\mathcal {M} _ {1}, \mathcal {M} _ {2}} ^ {\pi_ {1}, \pi_ {2}} (s, a) := \mathbb {E} _ {s ^ {\prime}, a ^ {\prime} \sim P _ {1}, \pi_ {1}} \left[ Q _ {\mathcal {M} _ {2}} ^ {\pi_ {2}} (s ^ {\prime}, a ^ {\prime}) \right] - \mathbb {E} _ {s ^ {\prime}, a ^ {\prime} \sim P _ {2}, \pi_ {2}} \left[ Q _ {\mathcal {M} _ {2}} ^ {\pi_ {2}} (s ^ {\prime}, a ^ {\prime}) \right] \\ = \underbrace {\mathbb {E} _ {s ^ {\prime} , a ^ {\prime} \sim P _ {1} , \pi_ {1}} \left[ Q _ {\mathcal {M} _ {2}} ^ {\pi_ {2}} \left(s ^ {\prime} , a ^ {\prime}\right) \right] - \mathbb {E} _ {s ^ {\prime} , a ^ {\prime} \sim P _ {1} , \pi_ {2}} \left[ Q _ {\mathcal {M} _ {2}} ^ {\pi_ {2}} \left(s ^ {\prime} , a ^ {\prime}\right) \right]} _ {(a)} \\ + \underbrace {\mathbb {E} _ {s ^ {\prime} , a ^ {\prime} \sim P _ {1} , \pi_ {2}} \left[ Q _ {\mathcal {M} _ {2}} ^ {\pi_ {2}} \left(s ^ {\prime} , a ^ {\prime}\right) \right] - \mathbb {E} _ {s ^ {\prime} , a ^ {\prime} \sim P _ {2} , \pi_ {2}} \left[ Q _ {\mathcal {M} _ {2}} ^ {\pi_ {2}} \left(s ^ {\prime} , a ^ {\prime}\right) \right]} _ {(b)}. \\ \end{array}
$$
For  $(a)$ , we have
$$
\begin{array}{l} (a) = \mathbb {E} _ {s ^ {\prime} \sim P _ {1}} \left[ \sum_ {a ^ {\prime}} \pi_ {1} (a ^ {\prime} | s ^ {\prime}) Q _ {\mathcal {M} _ {2}} ^ {\pi_ {2}} (s ^ {\prime}, a ^ {\prime}) - \pi_ {2} (a ^ {\prime} | s ^ {\prime}) Q _ {\mathcal {M} _ {2}} ^ {\pi_ {2}} (s ^ {\prime}, a ^ {\prime}) \right] \\ \leq \mathbb {E} _ {s ^ {\prime} \sim P _ {1}} \left[ \sum_ {a ^ {\prime}} \left| \pi_ {1} \left(a ^ {\prime} \mid s ^ {\prime}\right) - \pi_ {2} \left(a ^ {\prime} \mid s ^ {\prime}\right) \right| \frac {r _ {\max}}{1 - \gamma} \right] \\ = \frac {r _ {\operatorname* {m a x}}}{1 - \gamma} \mathbb {E} _ {s ^ {\prime} \sim P _ {1}} \left[ \sum_ {a ^ {\prime}} | \pi_ {1} (a ^ {\prime} | s ^ {\prime}) - \pi_ {2} (a ^ {\prime} | s ^ {\prime}) | \right] \\ = \frac {2 r _ {\mathrm {m a x}}}{1 - \gamma} \mathbb {E} _ {s ^ {\prime} \sim P _ {1}} \left[ D _ {T V} \left(\pi_ {1} (\cdot | s ^ {\prime}) \parallel \pi_ {2} (\cdot | s ^ {\prime})\right) \right]. \\ \end{array}
$$
For  $(b)$ , we have
$$
\begin{array}{l} (b) = \mathbb {E} _ {s ^ {\prime}, a ^ {\prime} \sim P _ {1}, \pi_ {2}} \left[ Q _ {\mathcal {M} _ {2}} ^ {\pi_ {2}} \left(s ^ {\prime}, a ^ {\prime}\right) \right] - \mathbb {E} _ {s ^ {\prime}, a ^ {\prime} \sim P _ {2}, \pi_ {2}} \left[ Q _ {\mathcal {M} _ {2}} ^ {\pi_ {2}} \left(s ^ {\prime}, a ^ {\prime}\right) \right] \\ \leq \left| \mathbb {E} _ {s ^ {\prime}, a ^ {\prime} \sim P _ {1}, \pi_ {2}} \left[ Q _ {\mathcal {M} _ {2}} ^ {\pi_ {2}} (s ^ {\prime}, a ^ {\prime}) \right] - \mathbb {E} _ {s ^ {\prime}, a ^ {\prime} \sim P _ {2}, \pi_ {2}} \left[ Q _ {\mathcal {M} _ {2}} ^ {\pi_ {2}} (s ^ {\prime}, a ^ {\prime}) \right] \right|. \\ \end{array}
$$
Adding these two bounds together yields the desired result.
D Detailed Environment Setting
D.1 Grid World
In the grid world environment, the agent obtains the X-Y coordination as the state and executes one of the four actions (Up, Down, Left, Right) at each time step. A non-zero reward 1.0 is provided only if the agent reaches the goal. Each episode terminates when the agent reaches the goal or the episode length of 256 is reached. The source domain and the target domain of the grid world are shown in Figure 9. For each algorithm, the agent interacts with the source and target domains for  $5e^{5}$  and  $5e^{4}$  steps, respectively.
Figure 9: The source domain (Left) and the target domain (Right) of the grid world environments.
D.2 Mujoco Environments
To investigate the performance of the algorithm thoroughly, we design eight environments based on four Mujoco [71] benchmarks from Gym [7] including HalfCheetah-v2, Ant-v4, Walker2D-v2, and Hopper-v2. For each benchmark, we propose two variants with kinematic shift or morphology shift. We run all experiments with the original environment as the source domain and the variation environment as the target domain. Detailed modifications of the environments are shown below, and the illustration of the environments is shown in Figure 10. For algorithms that access interactions with both domains, the agent interacts with the source and target domains for  $10^{6}$  and  $10^{5}$  steps, respectively.
Detailed modifications of the environments with kinematic shifts are shown below:
HalfCheetah - broken back thigh: We modify the rotation range of the joint on the thigh of the back leg from  $[-0.52, 1.05]$  to  $[-0.0052, 0.0105]$ .
Figure 10: Illustration of all environments, including all source domains (Top), all target domains with kinematic shifts (Middle), and all target domains with morphology shifts (Bottom).
Ant - broken hips: We modify the rotation range of the joints on the hip of leg 1 and leg 2 from  $[-30, 30]$  to  $[-0.3, 0.3]$ .
Walker - broken right foot: We modify the rotation range of the joint on the foot of the right leg from  $[-45, 45]$  to  $[-0.45, 0.45]$ .
Hopper - broken joints: We modify the rotation range of the joint on the head from  $[-150, 0]$  to  $[-0.15, 0]$  and the joint on foot from  $[-45, 45]$  to  $[-18, 18]$ .
Detailed modifications of the environments with morphology shifts are shown below:
HalfCheetah - no thighs: We modify the size of both thighs. Detailed modifications of the xml file are:
<geom fromto="0 0 0 -0.0001 0 -0.0001" name="bthigh" size="0.046" type = "capsule"/> <body name="bshin" pos="-0.0001 0 -0.0001">
<geom fromto="0 0 0 0.0001 0 0.0001" name="fthigh" size="0.046" type="capsule”/>   
<body name  $=$  "fshin"pos  $=$  "0.0001 0 0.0001">
Ant - short feet: We modify the size of feet on leg 1 and leg 2. Detailed modifications of the xml file are:
<geom fromto="0.0 0.0 0.0 0.1 0.1 0.0" name="left_ankle_geom" size="0.08" type="capsule"/>   
<geom fromto="0.0 0.0 0.0 -0.1 0.1 0.0" name="right_ankle_geom" size="0.08" type="capsule"/>
Walker - no right thigh: We modify the size of thigh on the right leg. Detailed modifications of the xml file are:
<body name="thigh" pos="0 0 1.05">
    <joint axis="0 -1 0" name="thigh_joint" pos="0 0 1.05" range="-150 0" type="hinge"/>
    <geom friction="0.9" fromto="0 0 1.05 0 0 1.045" name="thigh_geom"
        size="0.05" type="capsule"/>
    <body name="leg" pos="0 0 0.35">
        <joint axis="0 -1 0" name="leg_joint" pos="0 0 1.045" range="-150 0" type="hinge"/>
        <geom friction="0.9" fromto="0 0 1.045 0 0 0.3" name="leg_geom"
            size="0.04" type="capsule"/>
    <body name="foot" pos="0.2 0 0">
</body>
<geom friction="0.9" fromto="0 0 1.045 0 0 0.3" name="leg_geom"
        size="0.04" type="capsule"/>
</body>
<geom friction="0.9" fromto="0 0 1.045 0 0 0.3" name="leg_geom"
        size="0.04" type="capsule"/>
8 <joint axis="0 -1 0" name="foot_joint" pos="0 0 0.3" range="-45 45" type="hinge"/>  
9 <geom friction="0.9" fromto=-0.0 0 0.3 0.2 0 0.3" name="foot_geom" size="0.06" type="capsule"/>  
10 </body>  
11 </body>  
12 </body>
Hopper - big head: We modify the size of the head. Detailed modifications of the xml file are:
<geom friction="0.9" fromto="0 0 1.45 0 0 1.05" name="torso_geom" size="0.125" type="capsule"/>
E Algorithms and Implementation Details
E.1 Implementation Details
The details of our algorithm and baseline methods are specified as follows:
SAC: We first specify the implementation of the shared backbone algorithm SAC utilized in all algorithms. The policy and the value function are two-layer MLP with 256 hidden units using ReLU activation. The learning rate is  $3e^{-4}$ . Discount  $\gamma$  is set as 0.99 in all environments. The temperature coefficient is fixed as 0.2. The batch size is 128. The smoothing coefficient of the target networks is 0.005. The training delay of the policy is set as 2. The replay buffer size is  $1e^6$ .
VGDF: We use a five-layer MLP with 200 units as the dynamics model using Swish activation following prior works [10, 30]. The ensemble size is 7. We set the data selection ratio  $\xi \%$  as  $25\%$  in the experiments shown in Section 6.1. For each probabilistic dynamics model  $T_{\phi_i}(s_{t + 1},r_t|s_t,a_t) = \mathcal{N}(\mu_{\phi_i}(s_t,a_t),\Sigma_{\phi_i}(s_t,a_t))$ ,  $i = 1,\dots ,M$ , we train the model by maximizing the objective:
$$
\begin{array}{l} J \left(\phi_ {i}\right) := \mathbb {E} _ {\left(s _ {t}, a _ {t}, r _ {t}, s _ {t + 1}\right)} \sim D _ {t a r} \left[ \left[ \mu_ {\phi_ {i}} \left(s _ {t}, a _ {t}\right) - \right. \right. \\ \left. \left(s _ {t + 1}, r _ {t}\right) \right] ^ {\top} \Sigma_ {\phi_ {i}} ^ {- 1} \left(s _ {t}, a _ {t}\right) \left[ \mu_ {\phi_ {i}} \left(s _ {t}, a _ {t}\right) - \left(s _ {t + 1}, r _ {t}\right) \right] + \log \det \Sigma_ {\phi_ {i}} \left(s _ {t}, a _ {t}\right) ]. \tag {13} \\ \end{array}
$$
The exploration policy is a two-layer MLP with 256 hidden units. We warm-start the algorithm by utilizing samples from both domains without selection for the first  $1e5$  steps in the source domain.
DARC: We follow the default configurations of the public implementation (https://github.com/google-research/google-research/tree/master/darc). The domain classifiers  $q_{\psi_{SA}}(s_t, a_t, s_{t+1})$ ,  $q_{\psi_{SA}}(s_t, a_t)$  are trained by maximizing the cross-entropy losses: