SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Kang Xu $^{1}$  2* Chenjia Bai $^{2\dagger}$  Xiaoteng Ma $^{3}$  Dong Wang $^{2}$  Bin Zhao $^{2}$
Zhen Wang4 Xuelong  $\mathbf{L}\mathbf{i}^{2,4}$  Wei Li
$^{1}$  Fudan University  $^{2}$  Shanghai Artificial Intelligence Laboratory  $^{3}$  Tsinghua University  $^{4}$  Northwestern Polytechnical University
Generalizing policies across different domains with dynamics mismatch poses a significant challenge in reinforcement learning. For example, a robot learns the policy in a simulator, but when it is deployed in the real world, the dynamics of the environment may be different. Given the source and target domain with dynamics mismatch, we consider the online dynamics adaptation problem, in which case the agent can access sufficient source domain data while online interactions with the target domain are limited. Existing research has attempted to solve the problem from the dynamics discrepancy perspective. In this work, we reveal the limitations of these methods and explore the problem from the value difference perspective via a novel insight on the value consistency across domains. Specifically, we present the Value-Guided Data Filtering (VGDF) algorithm, which selectively shares transitions from the source domain based on the proximity of paired value targets across the two domains. Empirical results on various environments with kinematic and morphology shifts demonstrate that our method achieves superior performance compared to prior approaches.
Reinforcement Learning (RL) has demonstrated the ability to train highly effective policies with complex behaviors through extensive interactions with the environment [62, 59, 2]. However, in many situations, extensive interactions are infeasible due to the data collection costs and the potential safety hazards associated with domains such as robotics [33] and medical treatments [54]. To address the issue, one approach is to interact with a surrogate environment, such as a simulator, and then transfer the learned policy to the original domain. However, an unbiased simulator may be unavailable due to the complex system dynamics or unexpected disturbances in the target scenario, leading to a dynamics mismatch. Such a mismatch is crucial for the sim-to-real problem in robotics [1, 38, 51] and may cause performance degradation of the learned policy in the target domain. In this work, we focus on the dynamics adaptation problem, where we aim to train a well-performing policy for the target domain, given the source domain with the dynamics mismatch.
Recent research has tackled the adaptation over dynamics mismatch through various techniques, such as domain randomization [56, 53, 45], system identification [77], or simulator calibration [8], that require domain knowledge or privileged access to the physical system. Other methods have explored
*Part of this work was done during Kang Xu's internship at Shanghai Artificial Intelligence Laboratory.
$^{\dagger}$  Correspondence to Chenjia Bai <baichenjia@pjlab.org.cn>, Wei Li <fd_liwei@fudan.edu.cn>.
Figure 1: Semantic illustration of main settings for dynamics adaptation problem. Methods in the first three categories require different assumptions, such as a wide range of source domains, demonstrations from the target domain, or a manipulable simulator. We focus on a more general setting, online dynamics adaptation, only requiring limited online interactions with the target domain.
the adaptation problem in specific scenarios, such as those with expert demonstrations [41, 32] or offline datasets [42, 49], while the effectiveness of these methods heavily depends on the optimality of demonstrations or the quality of the datasets. In contrast to these works, we consider a more general setting called online dynamics adaptation, where the agent can access sufficient source domain data and a limited number of online interactions with the target domain. We compare the settings for the dynamics adaptation problem in Figure 1.
To address the online dynamics adaptation problem, prior works mainly focus on the single-step dynamics discrepancy and practically eliminating the gap via different ways [17, 14]. However, we empirically demonstrate the limitation of the methods through a motivation example, suggesting their effectiveness heavily relies on strong assumptions about the transferability of paired domains. Theoretically, we formulate the performance bound of the learned policy with respect to the dynamics discrepancy term, which provides an explicit interpretation of the results. To address the problem, we focus on the value discrepancy between paired transitions across domains, motivated by the key idea: the transitions with consistent value targets can be seen as equivalent for policy adaptation. Based on the insight, we proposed a simple yet efficient algorithm called Value-Guided Data Filtering (VGDF) for online dynamics adaptation via selective data sharing. Specifically, we use a learned target domain dynamics model to obtain paired transitions based on the source domain state-action pair. The transitions are shared from the source to the target domain only if the value targets of the imagined target domain transition and that of the source domain transition are close. Compared to previous methods that utilize the single-step dynamics gap, our method measures value discrepancies to capture long-term differences between two domains for better adaptation.
Our contributions can be summarized as follows: 1) We reveal the limitations of prior dynamics-based methods and propose the value discrepancy perspective with theoretical analysis. 2) To provide a practical instantiation, we propose VGDF for online dynamics adaptation via selective data sharing. 3) We extend VGDF to a more practical setting with an offline source domain dataset and propose a variant algorithm motivated by novel theoretical results. 4) We empirically demonstrate the superior performance of our method given significant dynamics shifts, including kinematics and morphology mismatch, compared to previous methods.
Domain adaptation in RL. Different from domain adaptation in supervised learning where different domains correspond to distinct data distributions [34], different domains in RL can differ in observation space [26], transition dynamics [56, 77, 17], embodiment [79, 43], or reward functions [16, 81, 57]. In this work, we focus on domain adaptation with dynamics discrepancies. Prior works utilizing meta RL [76, 48, 55], domain randomization [56, 53, 45], and system identification [80, 77, 15, 74] all assume the access to the distribution of training environments and rely on the hypothesis that the source and target domains are drawn from the same distribution. Another line of work has proposed to handle domain adaptation given expert demonstrations from the target domain [41, 32, 27]. These approaches align the state visitation distributions of the trained policy in the source domain to the distribution of the expert demonstrations in the target domain through state-action correspondences [79] or imitation learning [28, 21, 72]. However, near-optimal demonstrations can be challenging to acquire in some tasks. More recent works have explored the dynamics adaptation given an offline dataset collected in the target domain [42, 49], while the performance of
the trained policy depends on the quality of the dataset [50]. Orthogonal to these settings, we focus on a general paradigm where a relatively small number of online interactions with the target domain are accessible.
Online dynamics adaptation. Given limited online interactions with the target domain, several works calibrate the dynamics of the source domain by adjusting the physical parameters of the simulator [8, 58, 15, 47], while they assume the access of a manipulable simulator. Action transformation methods correct the transitions collected in the source domain by learning dynamics models of the two domains [25, 14, 78]. However, the learned model can be inaccurate, which results in model exploitation and performance degradation [30, 31]. Furthermore, the work that compensates the dynamics gap by modifying the reward function [17] is practical only if the policy that performs well in both domains exists. Instead, we do not assume the dynamics-agnostic policy exists and demonstrate the effectiveness of our method when such an assumption does not hold.
Knowledge transfer in RL. Knowledge transfer has been proposed to reuse the knowledge from other tasks to boost the training for the current task [69, 37]. The transferred knowledge can be modules (e.g., policy) [52, 9, 4], representations [5], and experiences [29, 39, 75, 68]. Our method is related to works transferring experiences. However, prior works focus on transferring between tasks with different reward functions instead of dynamics. When the dynamics changes, the direct adoption of commonly used temporal difference error [63] or advantage function [60] in previous works [29, 39, 68] would be inappropriate due to the shifted transition probabilities across domains. In contrast, we introduce novel measurements to evaluate the usefulness of the source domain transitions to tackle the dynamics shift problem specifically.
Theories on learning with dynamics mismatch. The performance guarantee of a policy trained with imaginary transitions from an inaccurate dynamics model has been analyzed in prior Dynasty [64, 65, 67] model-based RL algorithms [44, 30, 61]. The theoretical results inspire us to formulate performance guarantees in the context of dynamics adaptation.
3 Preliminaries and Problem Statement
We consider two infinite-horizon Markov Decision Processes (MDP)  $\mathcal{M}_{src} \coloneqq (\mathcal{S}, \mathcal{A}, P_{src}, r, \gamma, \rho_0)$  and  $\mathcal{M}_{tar} \coloneqq (\mathcal{S}, \mathcal{A}, P_{tar}, r, \gamma, \rho_0)$  for the source domain and the target domain, respectively. The two domains share the same state space  $\mathcal{S}$ , action space  $\mathcal{A}$ , reward function  $r: \mathcal{S} \times \mathcal{A} \to \mathbb{R}$  with range  $[0, r_{\max}]$ , discount factor  $\gamma \in [0, 1)$ , and the initial state distribution  $\rho_0: \mathcal{S} \to [0, 1]$ . The two domains differ on the transition probabilities, i.e.,  $P_{src}(s'|s, a)$  and  $P_{tar}(s'|s, a)$ .
We define the probability that a policy  $\pi$  encounters state  $s$  at the time step  $t$  in MDP  $\mathcal{M}$  as  $\mathrm{P}_{\mathcal{M},t}^{\pi}(s)$ . We denote the normalized probability that a policy  $\pi$  encounters state  $s$  in  $\mathcal{M}$  as  $\nu_{\mathcal{M}}^{\pi}(s) := (1 - \gamma) \sum_{t=0}^{\infty} \gamma^{t} \mathrm{P}_{\mathcal{M},t}^{\pi}(s)$ , and the normalized probability that a policy encounters state-action pair  $(s, a)$  in  $\mathcal{M}$  is  $\rho_{\mathcal{M}}^{\pi}(s, a) := (1 - \gamma) \sum_{t=0}^{\infty} \gamma^{t} \mathrm{P}_{\mathcal{M},t}^{\pi}(s) \pi(a|s)$ . The performance of a policy  $\pi$  in  $\mathcal{M}$  as is formally defined as  $\eta_{\mathcal{M}}(\pi) := \mathbb{E}_{s, a \sim \rho_{\mathcal{M}}^{\pi}}[r(s, a)]$ .
We focus on the online dynamics adaptation problem where limited online interactions with the target domain are accessible, which can be defined as follows:
Definition 3.1. (Online Dynamics Adaptation) Given source domain  $\mathcal{M}_{src}$  and target domain  $\mathcal{M}_{tar}$  with different dynamics, we assume sufficient data from the source domain (online or offline) and a relatively small number of online interactions with  $\mathcal{M}_{tar}$  (e.g.,  $\Gamma := \frac{\# \text{source domain data}}{\# \text{target domain data}} = 10$ ), hoping to obtain a near-optimal policy  $\pi$  concerning the target domain  $\mathcal{M}_{tar}$ .
The prior work [17] also focuses on the online dynamics adaptation problem with online source domain interactions. The proposed algorithm DARC estimates the dynamics discrepancy via learned domain classifiers and further introduces a reward correction (i.e.,  $\Delta r(s, a, s') \approx \log(P_{tar}(s'|s, a) / P_{src}(s'|s, a))$ ) to optimize policy together with the task reward  $r$  (i.e.,  $r(s, a) + \Delta r(s, a, s'))$ , discouraging the agent from dynamics-inconsistent behaviors in the source domain.
4 Guaranteeing Policy Performance from a Value Discrepancy Perspective
In this section, we will first present an example demonstrating the limitation of the prior method considering the dynamics discrepancy. Following that, we provide a theoretical analysis of the
Target Domain
(a)
(b)
Figure 2: The illustrations and results of the motivation experiment. (a) Illustration of the source and target domains in the grid world environment. The red dot and green square represent the agent and goal, respectively. (b) Visualization of the state visitation in both domains. The darker color suggests higher visitation probabilities. Our method guides the agent to reach regions with high target domain values while the agent trained by DARC is stuck in the room. (c) Visualization of the learned Q tables. Four triangles represent four actions; the darker color suggests a higher value estimation. Our method learns the optimal Q table whose greedy policy leads the agent to the goal of the target domain, while DARC fails due to pessimistic values of the crucial state-action pairs with dynamics mismatch.
dynamics-based method to provide an interpretation of the experiment results. Finally, we introduce a novel perspective on value discrepancies across domains for the online dynamics adaptation problem.
4.1 Motivation Example
We start with a 2D grid world task shown in Figure 2 (a), where the agent represented by the red dot needs to navigate to the green square representing the goal. We design source and target domains with different layouts and train a policy to reach the goal successfully in the target domain. We investigate the performance of DARC [17] that trains the policy with dynamics-guided reward correction and our proposed method (Section 5), using tabular  $Q$ -learning [73] as the backbone for all methods. Detailed environment settings are shown in Appendix D.
As the empirical state visitations and the learned  $Q$  tables show in Figure 2, DARC is stuck in the room and fails to obtain near-optimal  $Q$ -values, leading to poor performance. Specifically, we circle out four positions where specific actions will lead to the states with a dynamics mismatch concerning the two domains. Due to the introduced reward correction on the source domain transitions with dynamics mismatch, DARC learns overly pessimistic value estimations of particular state-action pairs, which hinders the agent from the optimal trajectory concerning the target domain. However, the values of the following inconsistent states, induced by the particular state-action pairs, are not significantly different concerning the target domain. The value difference quantifies the discrepancy of the long-term behaviors rather than single-step dynamics. Motivated by the value discrepancy perspective, our proposed method (Section 5.1) demonstrates superior performance.
4.2 Theoretical Interpretations and Value Discrepancy Perspective
To provide rigorous interpretations for the results, we derive a performance guarantee for the dynamics-guided methods, which mainly build on the theories proposed in prior methods [30, 17].
Theorem 4.1. (Performance bound controlled by dynamics discrepancy.) Denote the source domain and target domain with different dynamics as  $\mathcal{M}_{src}$  and  $\mathcal{M}_{tar}$ , respectively. We have the performance difference of any policy  $\pi$  evaluated under  $\mathcal{M}_{src}$  and  $\mathcal{M}_{tar}$  be bounded as below,
$$
\eta_ {\mathcal {M} _ {t a r}} (\pi) \geq \eta_ {\mathcal {M} _ {s r c}} (\pi) - \frac {2 \gamma r _ {\operatorname* {m a x}}}{(1 - \gamma) ^ {2}} \cdot \underbrace {\mathbb {E} _ {\rho_ {s r c} ^ {\pi}} \left[ D _ {\mathrm {T V}} \left(P _ {s r c} (\cdot | s , a) \| P _ {t a r} (\cdot | s , a)\right) \right]} _ {\text {(1)}}. \tag {1}
$$
(a) dynamics discrepancy
The proof of Theorem 4.1 is given in Appendix B. We observe that the derived performance bound in (1) is controlled by the dynamics discrepancy term (a). Intuitively, the performance difference would be minor when the dynamics discrepancy between the two domains is negligible. DARC [17] applies
the Pinsker's inequality [13] and derives the following form:
$$
\begin{array}{l} \eta_ {\mathcal {M} _ {t a r}} (\pi) \geq \eta_ {\mathcal {M} _ {s r c}} (\pi) - \frac {\gamma r _ {\mathrm {m a x}}}{(1 - \gamma) ^ {2}} \cdot \sqrt {2 \mathbb {E} _ {\rho_ {s r c} ^ {\pi}} [ D _ {\mathrm {K L}} (P _ {s r c} (\cdot | s , a) \| P _ {t a r} (\cdot | s , a)) ]} \\ = \eta_ {\mathcal {M} _ {s r c}} (\pi) + \frac {\gamma r _ {\operatorname* {m a x}}}{(1 - \gamma) ^ {2}} \cdot \sqrt {2 \mathbb {E} _ {\rho_ {s r c} ^ {\pi} , P _ {s r c}} [ \log (P _ {t a r} (s ^ {\prime} | s , a) / P _ {s r c} (s ^ {\prime} | s , a)) ]}. \tag {2} \\ \end{array}
$$
Based on the result in (2), DARC optimizes the policy by converting the second term in RHS to a reward correction (i.e.,  $\Delta r \coloneqq \log(P_{tar}(s'|s, a) / P_{src}(s'|s, a))$ ), leading to the dynamics discrepancy-based adaptation. However, given the transition from the source domain (i.e.,  $P_{src}(s'|s, a) \approx 1$ ), the reward correction will lead to significant penalty (i.e.,  $\log(P_{tar}(s'|s, a) / P_{src}(s'|s, a)) \ll 0$ ) if the likelihood estimation of the transition concerning the target domain is low (i.e.,  $P_{tar}(s'|s, a) \approx 0$ ). Consequently, the value estimation of the transition with dynamics mismatch tends to be overly pessimistic as shown in Figure 2 (c), which hinders learning an effective policy concerning the target domain.
Instead of myopically considering the single-step dynamics mismatch, we claim that the transitions with significant dynamics mismatch can be equivalent concerning the value estimations that evaluate the long-term behaviors. Due to the dynamics shift across domains, a state-action pair (i.e.,  $(s,a)$ ) would lead to two different next-states (i.e.,  $s_{src}^{\prime}$ ,  $s_{tar}^{\prime}$ ), the paired transitions are nearly equivalent for temporal different learning if the induced value estimations are close (i.e.,  $|V(s_{src}^{\prime}) - V(s_{tar}^{\prime})| \leq \epsilon$ ). Motivated by this, we derive a performance guarantee from the value difference perspective.
Theorem 4.2. (Performance bound controlled by value difference.) Denote source domain and target domain as  $\mathcal{M}_{src}$  and  $\mathcal{M}_{tar}$ , respectively. We have the performance guarantee of any policy  $\pi$  over the two MDPs:
$$
\eta_ {\mathcal {M} _ {t a r}} (\pi) \geq \eta_ {\mathcal {M} _ {s r c}} (\pi) - \frac {\gamma}{1 - \gamma} \cdot \underbrace {\mathbb {E} _ {\rho_ {\mathcal {M} _ {s r c}} ^ {\pi}} \left[ \left| \mathbb {E} _ {P _ {s r c}} \left[ V _ {\mathcal {M} _ {t a r}} ^ {\pi} \left(s ^ {\prime}\right) \right] - \mathbb {E} _ {P _ {t a r}} \left[ V _ {\mathcal {M} _ {t a r}} ^ {\pi} \left(s ^ {\prime}\right) \right] \right| \right]} _ {(a): v a l u e d i f f e r e n c e}. \tag {3}
$$
The proof of Theorem 4.2 is given in Appendix B. The value difference term provides a novel perspective: the performance can be guaranteed if the transitions from the source domain lead to consistent value targets in the target domain. The result further highlights the value consistency perspective for the online dynamics adaptation problem.
5 Value-Guided Data Filtering
In this section, we propose Value-Guided Data Filtering (VGDF), a simple yet efficient algorithm for online domain adaptation via selective data sharing. Then we introduce the setting with offline source domain data and a variant of VGDF based on novel theoretical results. The pseudocodes are shown in Appendix A, and the illustration of VGDF is shown in Figure 3.
5.1 Dynamics Adaptation by Selective Data Sharing
Inspired by the performance bound proposed in Theorem 4.2, we can guarantee the policy performance by controlling the value difference term in (3). As discussed in Section 4.2, the paired transitions concerning two domains, induced by the same state-action pair, can be regarded as equivalent for temporal difference learning when the corresponding values are close. Thus, we propose to select source domain transitions with minor value discrepancies for dynamics adaptation.
To select rational transitions from the source domain, we need to compare the value differences of paired transitions based on the same source domain state-action pair  $(s_{src}, a_{src})$ . Formally, given a state-action pair  $(s_{src}, a_{src})$  from the source domain, our objective is to estimate whether the value-difference between  $s_{tar}'$  and  $s_{src}'$  is sufficiently small, i.e.,
$$
\Delta \left(s _ {s r c}, a _ {s r c}\right) := 1 1 \left(\left| V _ {\mathcal {M} _ {t a r}} ^ {\pi} \left(s _ {t a r} ^ {\prime}\right) - V _ {\mathcal {M} _ {t a r}} ^ {\pi} \left(s _ {s r c} ^ {\prime}\right) \right| \leq \epsilon\right), \tag {4}
$$
where  $s_{tar}^{\prime} \sim P_{tar}(\cdot |s_{src},a_{src})$ ,  $s_{src}^{\prime} \sim P_{src}(\cdot |s_{src},a_{src})$ ,  $\mathbb{1}$  denotes the indicator function and  $\epsilon$  can be a predefined threshold.
To obtain  $\Delta(s_{src}, a_{src})$ , we need to perform policy evaluation over the states to obtain the value estimations given the paired next states (i.e.,  $s_{src}'$ ,  $s_{tar}'$ ), as formulated in Eq. (4). Monte Carlo (MC)
Figure 3: Semantic illustration of VGDF. We tackle online dynamics adaptation by selectively sharing the source domain data, and the RL denotes any off-the-shelf off-policy RL algorithm.
evaluation can provide unbiased values by rolling the policy starting from specific states [66]. However, since the environment is not manipulable, we cannot perform MC evaluation from arbitrary states. Thus, we propose to use an estimated value function for policy evaluation. In this work, we adopt the Fitted Q Evaluation (FQE) [46] that is widely used in off-policy RL algorithms [40, 23, 24]. Specifically, we utilize a learned Q function  $Q_{\theta}:S\times A\to \mathbb{R}$  for evaluation.
Furthermore, one problem is that the corresponding target domain next state  $s_{tar}^{\prime}$  induced by  $(s_{src}, a_{src})$  is unavailable in practice. To achieve this, we train a dynamics model with the collected data from the target domain. Following prior works [36, 10], we employ an ensemble of Gaussian dynamics models  $\{T_{\phi_i}(s'|s, a)\}_{i=1}^{M}$ , in an attempt to capture the epistemic uncertainty due to the insufficient target domain samples. Given the source domain state-action pair  $(s_{src}, a_{src})$ , we generate an ensemble of fictitious states and obtain the corresponding values for each state-action pair, which we call fictitious value ensemble (FVE)  $\mathcal{Q}_{tar}^{\pi}(s_{src}, a_{src})$ :
$$
\mathcal {Q} _ {t a r} ^ {\pi} \left(s _ {s r c}, a _ {s r c}\right) := \left\{Q _ {\theta} \left(s _ {i} ^ {\prime}, a _ {i} ^ {\prime}\right) \mid_ {s _ {i} ^ {\prime}} \sim T _ {\phi_ {i}} \left(\cdot \mid s _ {s r c}, a _ {s r c}\right), a _ {i} ^ {\prime} \sim \pi \left(\cdot \mid s _ {i} ^ {\prime}\right) \right\} _ {i = 1} ^ {M}. \tag {5}
$$
In practice, the choice of  $\epsilon$  in Eq. (4) is also nontrivial due to task-specific scales of the values and the non-stationary value function during training. We replace the absolute value difference with the likelihood estimation to address the problem. Specifically, we construct a Gaussian distribution with the mean and variance of FVE denoted as  $\mathcal{N}(\mathrm{Mean}(\mathcal{Q}_{tar}^{\pi}(s_{src},a_{src})),\mathrm{Var}(\mathcal{Q}_{tar}^{\pi}(s_{src},a_{src})))$ . Estimating the value of the source domain state as  $V_{tar}^{\pi}(s_{src}')\coloneqq Q_{\theta}(s_{src}',a_{src}')|_{a_{src}'}\sim \pi (\cdot |s_{src}')$ , we introduce Fictitious Value Proximity (FVP) representing the likelihood of the source domain state value in the distribution: