SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} J \left(\psi_ {S A S}\right) := \mathbb {E} _ {\left(s _ {t}, a _ {t}, s _ {t + 1}\right) \sim D _ {t a r}} \left[ \log q _ {\psi_ {S A S}} \left(t a r \mid s _ {t}, a _ {t}, s _ {t + 1}\right) \right] \\ + \mathbb {E} _ {\left(s _ {t}, a _ {t}, s _ {t + 1}\right) \sim D _ {s r c}} \left[ \log \left(1 - q _ {\psi_ {S A S}} \left(t a r \mid s _ {t}, a _ {t}, s _ {t + 1}\right)\right) \right], \\ \end{array}
$$
$$
J (\psi_ {S A}) := \mathbb {E} _ {(s _ {t}, a _ {t}) \sim D _ {t a r}} \left[ \log q _ {\psi_ {S A}} (t a r | s _ {t}, a _ {t}) \right] + \mathbb {E} _ {(s _ {t}, a _ {t}) \sim D _ {s r c}} \left[ \log \left(1 - q _ {\psi_ {S A}} (t a r | s _ {t}, a _ {t})\right) \right].
$$
Following the original implementation, we use the standard Gaussian noise for the domain classifier training. During training, a reward correction  $\Delta r(s_t, a_t)$  is augmented to the original reward  $r(s_t, a_t)$  of each source domain transition, i.e.  $\tilde{r}(s_t, a_t) := r(s_t, a_t) + \Delta r(s_t, a_t)$ . The reward correction is calculated by:
$$
\Delta r (s _ {t}, a _ {t}) := \log \frac {q _ {\psi_ {S A S}} (t a r | s , a , s ^ {\prime})}{q _ {\psi_ {S A S}} (s r c | s , a , s ^ {\prime})} \frac {q _ {\psi_ {S A}} (s r c | s , a)}{q _ {\psi_ {S A}} (t a r | s , a)}.
$$
We warm-start the algorithm by training with samples from both domains for the first  $10^{5}$  steps following the original implementation.
GARAT: We use the author implementation with default configurations (Supplemental in https://proceedings.neurips.cc/paper/2020/bit/28f248e9279ac845995c4e9f8af35c2b-Abstract.html). We add the XML files of our customized environments to r1_gat/environments/ assets/folder. We limit the extra interactions with the grounded source environments as  $10^{5}$  for fair comparisons with other algorithms.
Table 2: Hyperparameters. "-" denotes the hyperparameter is not used in the algorithm. "←" denotes the same choice as the algorithm in the first column.
Hyperparameters	VGDF	DARC	GARAT	IW Clip	Finetune
Hidden layers (Policy)	2	←	←	←	←
Hidden units per layer (Policy)	256	←	←	←	←
Hidden layers (Value)	2	←	←	←	←
Hidden units per layer (Value)	256	←	←	←	←
Hidden layers (Classifier)	-	2	-	2	-
Hidden units per layer (Classifier)	-	256	-	256	-
Hidden layers (Dynamics model)	5	-	-	-	-
Hidden units per layer (Dynamics model)	200	-	-	-	-
Ensemble size	7	-	-	-	-
Learning rate	3e-4	←	←	←	←
Batch size	128	←	←	←	←
Fixed temperature coefficient	0.2	←	←	←	←
Target smoothing coefficient	0.005	←	←	←	←
Policy training delay	2	←	←	←	←
Buffer size	1e6	←	←	←	←
Data selection ratio ξ%	25%	-	-	-	-
Warm-start steps	1e5	1e5	-	1e5	-
Importance weight clipping range	-	-	-	[1e-4, 1]	-
Interactions with grounded src environment	-	-	1e5	-	-
Importance Weighting Clip (IW Clip): We use the domain classifiers same as DARC to calculate the importance weight  $w(s, a, s')$ . The importance weighting is calculated by:
$$
w (s, a, s ^ {\prime}) := \frac {P _ {t a r} (s ^ {\prime} | s , a)}{P _ {s r c} (s ^ {\prime} | s , a)} \approx \frac {q _ {\psi_ {S A S}} (t a r | s , a , s ^ {\prime})}{q _ {\psi_ {S A S}} (s r c | s , a , s ^ {\prime})} \frac {q _ {\psi_ {S A}} (s r c | s , a)}{q _ {\psi_ {S A}} (t a r | s , a)},
$$
where  $q_{\psi_{SAS}}$  and  $q_{\psi_{SA}}$  are the domain classifiers proposed in [17]. We use the importance weighing to reweight the value training with source domain samples. Specifically,
$$
\theta \gets \underset {\theta} {\arg \min} \frac {1}{2} \mathbb {E} _ {(s, a, r, s ^ {\prime}) \sim D _ {s r c}} \left[ w (s, a, s ^ {\prime}) (Q _ {\theta} - \mathcal {T} Q _ {\theta}) ^ {2} \right].
$$
To stabilize training, we clip the importance weight between  $[1e^{-4}, 1]$ , same as the prior work [49].
Finetune: We first train a policy in the source domain with  $10^{6}$  steps. Then we transfer the policy to the target domain and further train the policy for  $10^{5}$  steps.
The detailed hyperparameters of all algorithms are listed in Table. 2, and we use the same hyperparameters across all environments.
E.2 Implementation Details of the Offline-Online Experiments
To evaluate the performance of our algorithm in the offline source online target setting, we use medium datasets from D4RL [20] for three environments (i.e., HalfCheetah, Hopper, Walker). We use the same source domain offline dataset for each environment's two different target domains. For the algorithms performing online learning using offline data (i.e., Symmetric sampling,  $H2O$ , VGDF + BC), we perform the online interactions with the target domain for  $10^{5}$  steps and use  $10^{6}$  source domain transitions, the training is repeated for 10 times per step in the target domain. The details of the methods are specified as follows:
Offline only: We directly transfer the policy learned through CQL [35] with the source domain offline dataset. For the CQL implementation, we follow the suggested configurations in a public CQL implementation (https://github.com/tinkoff-ai/CORL). We perform training for  $10^{6}$  steps with the offline dataset and report the zero-shot performance of the learned policy in the target domain.
Symmetric sampling [3]: We perform the value function training by combining CQL optimization (with offline transitions) and SAC optimization (with online transitions). For each training step, we
sample  $50\%$  of the data from the target domain replay buffer and the remaining  $50\%$  from the source domain offline dataset. The CQL and SAC loss is computed with the corresponding transitions.
H2O [49]: We follow the original implementation that learns the classifiers to estimate the dynamics discrepancy across domains and perform the clipped importance weighting on the CQL loss on the source domain data. Same as Symmetric sampling, we repeat the training for 10 times per step in the target domain.
$\mathbf{VGDF} + \mathbf{BC}$ : We adapt VGDF to the Offline-Online setting by simply integrating the behavior cloning loss following (10). The training is repeated for 10 times per step with the target domain the same as the baseline methods. For the trade-off between the policy gradient and behavior cloning, we use the value-normalized regularization following the  $TD3 + BC$  [22] work and set the constant  $\alpha$  as 5. Furthermore, we remove the exploration policy proposed in Section 5.1 since the online access to the source domain is no longer available in the offline-online setting.
F.1 Quantifying Dynamics Shifts via FVP
In this section, we investigate whether the estimation of the value differences can quantify the difference across domains. Specifically, in different target domains of the same source domain, we demonstrate the estimation of FVP in two target domains. As the results show in Figure 11, the FVP differs in environments with different dynamics shifts (Kinematic or morphology). We observe that the FVP values in two target domains gradually approach each other in three out of four environments (HalfCheetah, Walker, Hopper), while the values in Ant remain relatively stationary. Furthermore, the FVP values in target domains with kinematic shifts are lower than those with morphology shifts across all four environments, which could result from the mismatched state space due to the limited joint ranges of robots in the target domain. Given the differences across different environments, we believe the FVP estimation could be used to quantify the domain differences.
Figure 11: Quantification analysis of the approximated FVP in all environments with different dynamics shifts. The dots are averaged values, and the error bars indicate the standard error across five runs.
F.2 Sensitivity to Ensemble Size
Figure 12: Performance of the variants with different ensemble size values  $M$ . The results validate that a smaller ensemble size is sufficient to achieve competitive asymptotic performance compared to the variant with a large ensemble size in most environments.
We have introduced the dynamics model ensemble to capture the epistemic uncertainty induced by the limited samples from the target domain. However, training the ensemble of the dynamics model takes extra computation resources. Unlike prior works in model-based RL [30, 61] that utilize the generated samples for training, we measure the value difference with the help of the generated samples. Therefore, we aim to investigate whether a smaller ensemble size is sufficient to achieve competitive asymptotic performance. Here we set the ensemble size as different values ( $M = 7$  in the original implementation) and run experiments in four environments. As the results show in Figure 12, variants with a small ensemble size (e.g.,  $M = 3$  or  $M = 5$ ) can achieve identical asymptotic performance compared to the variant with a large ensemble size (e.g.,  $M = 7$ ) in three out of four environments.
F.3 What about Importance Weighting via FVP instead of Rejection Sampling?
Figure 13: Performance of the variants with rejection sampling or importance weighting technique. The results demonstrate that the original algorithm using rejection sampling outperforms the variant using importance weighting via FVP in almost all environments.
In the case of data selection based on the estimated FVP (fictitious value proximity in Eq. (6), one may wonder about using importance weighting via the FVP rather than rejection sampling, which might be sample-inefficient due to the discarded partial data. Here we implement a variant of our algorithm that performs importance weighting with the estimated fictitious value proximity. Specifically, we train the value functions following:
$$
\begin{array}{l} \theta_ {i = 1, 2} \leftarrow \arg \min _ {\theta_ {i}} \frac {1}{2 B} \sum_ {\{(s, a, r, s ^ {\prime}) \} _ {t a r} ^ {B}} \left[ (Q _ {\theta_ {i}} - \mathcal {T} Q _ {\theta_ {i}}) ^ {2} \right] + \\ \frac {1}{2 B} \sum_ {\{(s, a, r, s ^ {\prime}) \} _ {s r c} ^ {B}} \left[ \frac {\Lambda (s , a , s ^ {\prime})}{\sum_ {\{s , a , s ^ {\prime} \} ^ {B}} \Lambda (s , a , s ^ {\prime})} (Q _ {\theta_ {i}} - \mathcal {T} Q _ {\theta_ {i}}) ^ {2} \right]. \\ \end{array}
$$
We compare the variant with the original algorithm using rejection sampling in all eight environments and demonstrate the results in Figure 13. The original algorithm using rejection sampling outperforms the variant with importance weighting in almost all environments. The accuracy of the value proximity depends on the generated state and the value function. Thus, the estimation of FVP could be biased due to the inaccurate dynamics models and value functions in the early training stage, in which case naively utilizing the source domain samples weighted by the FVP can harm the policy performance concerning the target domain. In contrast, rejection sampling that only utilizes a small portion of source domain samples alleviates the negative effect of the source domain samples.
F.4 What about Data Filtering via Value instead of FVP?
Prior works have examined sharing data across tasks with different reward functions rather than dynamics [75]. To investigate whether selectively sharing data with a high Q value can address the
Figure 14: Performance of the variants that employ data filtering based on Value or FVP. The results demonstrate that the original algorithm outperforms the variant using data filtering via Value in four of eight environments.
Figure 15: Comparison with the variant performing data filtering based on estimated dynamics discrepancies. The results demonstrate that the original algorithm outperforms the variant using data filtering via Value in four of eight environments, validating the effect of the value consistency.
online dynamics adaptation problem, we propose a variant of our algorithm that shares partial data with a relatively high Q value from the source domain. Specifically, we train the value functions following:
$$
\begin{array}{l} \theta_ {i = 1, 2} \gets \underset {\theta_ {i}} {\arg \min} \frac {1}{2 B} \sum_ {\{(s, a, r, s ^ {\prime}) \} _ {t a r} ^ {B}} \left[ (Q _ {\theta_ {i}} - \mathcal {T} Q _ {\theta_ {i}}) ^ {2} \right] + \\ \frac{1}{[2B\cdot\xi\%]}\sum_{\{(s,a,r,s^{\prime})\}_{src}^{B}}\left[\mathbb{1}\left(Q_{\theta_{i}}(s,a) > Q_{\xi \%}\right)\left(Q_{\theta_{i}} - \mathcal{T}Q_{\theta_{i}}\right)^{2}\right], \\ \end{array}
$$
where  $Q_{\xi \%}$  is the top  $\xi$ -quantile Q value of a batch of source domain samples. We set  $\xi \%$  as  $25\%$ , the same as our implementation. We compare the variant with the original algorithm in all eight environments and demonstrate the results in Figure 14. The results demonstrate that the original algorithm outperforms the variant using data filtering via value in four of eight environments. Due to the dynamics mismatch, a state-action pair from the source domain will lead to inconsistent states concerning two domains. Therefore, directly utilizing the transitions with high Q value without considering the consistency of the next state would provide a counterfactual value target for the state-action pair, which can result in an improper value estimation for learning.
F.5 Comparison with Dynamics-guided Data Filtering
To investigate the effect of value consistency, we perform the ablation study by comparing VGDF to a variant that shares partial data based on dynamics discrepancies, i.e., Dynamics-guided Data Filtering (DGDF). Specifically, we estimate the dynamics discrepancy via the learned classifiers
Table 3: Results in PyBullet environments. We evaluate the algorithms via the performance of the learned policy in the target domain and report the mean and std of the results across five runs with different random seeds. (# source, # target) denotes the number of source domain data versus the number of target domain data. HC and HP denote HalfCheetah and Hopper, respectively.
DARC	Finetune	VGDF	DARC	Finetune	VGDF
(# source, # target)	200k, 20k	1M, 20k	200k, 20k	1M, 100k	1M, 100k	1M, 100k
PyBullet - HC	304 ± 211	653 ± 51	770 ±203	679 ± 131	678 ± 38	808 ±89
PyBullet - HP	73 ± 20	240 ± 189	957 ±39	99 ± 20	869 ± 32	1006 ±2
following the prior works [17, 49]. Same as VGDF, we share the source domain transitions whose estimated dynamics difference is smaller than the quantile value. We set the selection ratios as  $25\%$ , the same as our implementation. The results demonstrate that the original algorithm outperforms the variant in three out of four environments, validating the superior effect of the value consistency compared to the dynamics discrepancy.
F.6 Extended Results in Pybullet Environments
To investigate the generality of VGDF, we perform additional experiments in PyBullet-HalfCheetah and PyBullet-Hopper from PyBullet environments [12] which utilize Bullet as the physical engine instead of Mujoco. We first provide the details of the dynamics gap in the environments. In both environments, we regard the original environments as the source domains. In PyBullet-Hopper, we devised the target domain by increasing the torso size from 0.05 to 0.15, to simulate the morphology change. In PyBullet-HalfCheetah, we constrain the joint range of the front thigh from  $[-1.5, 0.8]$  to  $[-1.5, 0.4]$ , and the joint range of the front shin from  $[-1.2, 1.1]$  to  $[-1.2, 0.1]$ , to simulate the broken joint scenario that is widely used in related works.
The results are shown in the Table 3, and we report the performance of all algorithms concerning different numbers of target domain samples. All results are averaged across five runs with different seeds. The results demonstrate that VGDF consistently outperforms baselines given different number of target domain data, demonstrating the generality of our method.