DARA: DYNAMICS-AWARE REWARD AUGMENTATION
Year: 2021

Abstract:
Offline reinforcement learning algorithms promise to be applicable in settings where a fixed dataset is available and no new experiment can be acquired. However, such formulation is inevitably offline-data-hungry and, in practice, collecting a large offline dataset for one specific task over one specific environment is also costly and laborious. In this paper, we thus 1) formulate the offline dynamics adaptation by using (source) offline data collected from another dynamics to relax the requirement for the extensive (target) offline data, 2) characterize the dynamics shift problem in which prior offline methods do not scale well, and 3) derive a simple dynamics-aware reward augmentation (DARA) framework from both modelfree and model-based offline settings. Specifically, DARA emphasizes learning from those source transition pairs that are adaptive for the target environment and mitigates the offline dynamics shift by characterizing state-action-next-state pairs instead of the typical state-action distribution sketched by prior offline RL methods. The experimental evaluation demonstrates that DARA, by augmenting rewards in the source offline dataset, can acquire an adaptive policy for the target environment and yet significantly reduce the requirement of target offline data. With only modest amounts of target offline data, our performance consistently outperforms the prior offline RL methods in both simulated and real-world tasks.

text: 1 INTRODUCTION
Offline reinforcement learning (RL) (Levine et al., 2020; Lange et al., 2012), the task of learning from the previously collected dataset, holds the promise of acquiring policies without any costly active interaction required in the standard online RL paradigm. However, we note that although the active trail-and-error (online exploration) is eliminated, the performance of offline RL method heavily relies on the amount of offline data that is used for training. As shown in Figure 1, the performance deteriorates dramatically as the amount of offline data decreases. A natural question therefore arises: can we reduce the amount of the (target) offline data without significantly affecting the final performance for the target task?
Bringing the idea from the transfer learning (Pan & Yang, 2010), we assume that we have access to another (source) offline dataset, hoping that we can leverage this dataset to compensate for the performance degradation caused by the reduced (target) offline dataset. In the offline setting, previous work (Siegel et al., 2020; Chebotar et al., 2021) has characterized the reward (goal) difference between the source and target, relying on the ”conflicting” or multi-goal offline dataset (Fu et al., 2020), while we focus on the relatively unexplored transition dynamics difference between the source dataset and the target environment. Meanwhile, we believe that this dynamics shift is not arbitrary in reality: in healthcare treatment, offline data for a particular patient is often limited, whereas we can obtain diagnostic data from other patients with the same case (same reward/goal) and there often exist individual differences between patients (source dataset with different transition dynamics). Careful treatment with respect to the individual differences is thus a crucial requirement.
Given source offline data, the main challenge is to cope with the transition dynamics difference, i.e., strictly tracking the state-action supported by the source offline data can not guarantee that the same
transition (state-action-next-state) can be achieved in the target environment. However, in the offline setting, such dynamics shift is not explicitly characterized by the previous offline RL methods, where they typically attribute the difficulty of learning from offline data to the state-action distribution shift (Chen & Jiang, 2019; Liu et al., 2018). The corresponding algorithms (Fujimoto et al., 2019; Abdolmaleki et al., 2018; Yu et al., 2020) that model the support of state-action distribution induced by the learned policy, will inevitably suffer from the transfer problem where dynamics shift happens.
Our approach is motivated by the well established connection between reward modification and dynamics adaptation (Kumar et al., 2020b; Eysenbach & Levine, 2019; Eysenbach et al., 2021), which indicates that, by modifying rewards, one can train a policy in one environment and make the learned policy to be suitable for another environment (with different dynamics). Thus, we propose to exploit the joint distribution of state-action-next-state: besides characterizing the state-action distribution shift, we additionally identify the dynamics (i.e., conditional distribution of next-state given state-action pair) shift and penalize the agent with a dynamics-aware reward modification. This modification aims to discourage the learning from these offline transitions that are likely in source but are unlikely in target. Unlike the concurrent work (Ball et al., 2021; Mitchell et al., 2021) paying attention to the offline domain generalization, we explicitly focus on the offline domain (dynamics) adaptation.
Our principal contribution in this work is the characterization of the dynamics shift in offline RL and the derivation of dynamics-aware reward augmentation (DARA) framework built on prior modelfree and model-based formulations. DARA is simple and general, can accommodate various offline RL methods, and can be implemented in just a few lines of code on top of dataloader at training. In our offline dynamics adaptation setting, we also release a dataset, including the Gym-MuJoCo tasks (Walker2d, Hopper and HalfCheetah), with dynamics (mass, joint) shift compared to D4RL, and a 12-DoF quadruped robot in both simulator and real-world. With only modest amounts of target offline data, we show that DARA-based offline methods can acquire an adaptive policy for the target tasks and achieve better performance compared to baselines in both simulated and real-world tasks.

heading: 2 RELATED WORK
text: Offline RL describes the setting in which a learner has access to only a fixed dataset of experience, while no interactive data collection is allowed during policy learning (Levine et al., 2020). Prior work commonly assumes that the offline experience is collected by some behavior policies on the same environment that the learned policy be deployed on. Thus, the main difficulty of such offline setting is the state-action distribution shift (Fujimoto et al., 2019; Liu et al., 2018). Algorithms address this issue by following the two main directions: the model-free and model-based offline RL.
Model-free methods for such setting typically fall under three categories: 1) Typical methods mitigate this problem by explicitly (Fujimoto et al., 2019; Kumar et al., 2019; Wu et al., 2019) or implicitly (Siegel et al., 2020; Peng et al., 2019; Abdolmaleki et al., 2018) constraining the learned policy away from OOD state-action pairs. 2) Conservative estimation based methods learn pessimistic value functions to prevent the overestimation (Kumar et al., 2020a; Xu et al., 2021). 3) Importance sampling based methods directly estimate the state-marginal importance ratio and obtain an unbiased value estimation (Zhang et al., 2020; Nachum & Dai, 2020; Nachum et al., 2019b).
Model-based methods typically eliminate the state-action distribution shift by incorporating a reward penalty, which relies on the uncertainty quantification of the learned dynamics (Kidambi et al., 2020; Yu et al., 2020). To remove this uncertainty estimation, Yu et al. (2021) learns conservative critic function by penalizing the values of the generated state-action pairs that are not in the offline dataset.
These methods, however, define their objective based on the state-action distribution shift, and ignore the potential dynamics shift between the fixed offline data and the target MDP. In contrast, we account for dynamics (state-action-next-state) shift and explicitly propose the dynamics aware reward augmentation. A counterpart, close to our work, is off-dynamics RL (Eysenbach et al., 2021), where they set up dynamics shift in the interactive environment while we focus on the offline setting.

heading: 3 PRELIMINARIES
text: We study RL in the framework of Markov decision processes (MDPs) specified by the tuple M := (S,A, r, T, ρ0, γ), where S and A denote the state and action spaces, r(s,a) ∈ [−Rmax, Rmax] is
the reward function, T (s′|s,a) is the transition dynamics, ρ0(s) is the initial state distribution, and γ is the discount factor. The goal in RL is to optimize a policy π(a|s) that maximizes the expected discounted return ηM (π) := Eτ∼pπM (τ) [ ∑∞ t=0 γ
tr(st,at)], where τ := (s0,a0, s1,a1, ...). We also define Q-values Q(s,a) := Eτ∼pπM (τ) [ ∑∞ t=0 γ
tr(st,at)|s0 = s,a0 = a], V-values V (s) := Ea∼π(a|s) [Q(s,a)], and the (unnormalized) state visitation distribution dπM (s) := ∑∞ t=0 γ
tP (s|π, M, t), where P (s|π,M, t) denotes the probability of reach state s at time t by running π in M . In the offline RL problem, we are provided with a static dataset D := {(s,a, r, s′)}, which consists of transition tuples from trajectories collected by running one or more behavioral policies, denoted by πb, on MDP M . With a slight abuse of notation, we write D = {(s,a, r, s′) ∼ dD(s)πb(a|s)r(s,a)T (s′|s,a)}, where the dD(s) denotes state-marginal distribution in D. In the offline setting, the goal is typically to learn the best possible policy using the fixed offline dataset.
Model-free RL algorithms based on dynamic programming typically perform policy iteration to find the optimal policy. Such methods iteratively conduct 1) policy improvement with GMQ := arg maxπ Es∼dπM (s),a∼π(a|s) [Q(s,a)] and 2) policy evaluation by iterating the Bellman equation Q(s,a) = BπMQ(s,a) := r(s,a) + γEs′∼T (s′|s,a),a′∼π(a′|s′) [Q(s′,a′)] over dπM (s)π(a|s). Given off-policy data D, we resort to 1) improvement with GDQ := arg maxπ Es∼dD(s),a∼π(a|s) [Q(s,a)] and 2) evaluation by iteratingQ(s,a) = BπDQ(s,a) := r(s,a)+γEs′∼T (s′|s,a),a′∼π(a′|s′) [Q(s′,a′)] over all (s,a) in D. Specifically, given any initial Q0, it iterates1
Policy improvement: πk+1 = GDQk, Policy evaluation: Qk+1 = Bπ k+1 D Q k. (1)
Model-free offline RL based on the above iteration suffers from the state-action distribution shift, i.e., policy evaluation BπkD Qk−1 may encounter unfamiliar state action regime that is not covered by the fixed offline dataset D, causing erroneous estimation of Qk. Policy improvement GDQk further exaggerates such error, biasing policy πk+1 towards out-of-distribution (OOD) actions with erroneously high Q-values. To address this distribution shift, prior works 1) explicitly constrain policy to be close to the behavior policy (Fujimoto et al., 2019; Kumar et al., 2019; Wu et al., 2019; Ghasemipour et al., 2021), introducing penalty αD(π(a|s), πb(a|s)) into GD or BπD in Equation 1:
GDQ = arg max π Es∼dD(s),a∼π(a|s) [Q(s,a)− αD(π(a|s), πb(a|s))] ,
BπDQ(s,a) = r(s,a) + γEs′∼T (s′|s,a),a′∼π(a′|s′) [Q(s′,a′)− αD(π(a′|s′), πb(a′|s′))] , (2)
where D is a divergence function between distributions over actions (e.g., MMD or KL divergence), or 2) train pessimistic value functions (Kumar et al., 2020a; Yu et al., 2021; Xu et al., 2021), penalizing Q-values at states in the offline dataset D for actions generated by the current policy π:
Q = arg min Q
Es∼dD(s),a∼π(a|s) [Q(s,a)] , s.t. Q = B π DQ. (3)
Model-based RL algorithms iteratively 1) model the transition dynamics T (s′|s,a), using the data collected in M : maxT̂ Es,a,s′∼dπM (s)π(a|s)T (s′|s,a)[log T̂ (s
′|s,a)], and 2) infer a policy π from the modeled M̂ = (S,A, r, T̂ , ρ0, γ), where we assume that r and ρ0 are known, maximizing ηM̂ (π) with a planner or the Dyna-style algorithms (Sutton, 1990). In this paper, we focus on the latter.
Model-based offline RL algorithms similarly suffer from OOD state-action (Kidambi et al., 2020; Cang et al., 2021) if we directly apply policy iteration over T̂ := maxT̂ Es,a,s′∼D[log T̂ (s
′|s,a)]. Like the conservative estimation approach described in Equation 3, recent conservative model-based offline RL methods provide the policy with a penalty for visiting states under the estimated T̂ where T̂ is likely to be incorrect. Taking u(s,a) as the oracle uncertainty (Yu et al., 2020) that provides a consistent estimate of the accuracy of model T̂ at (s,a), we can modify the reward function to obtain a conservative MDP: M̂c = (S,A, r − αu, T̂ , ρ0, γ), then learn a policy π by maximizing ηM̂c(π).

heading: 4 PROBLEM FORMULATION
text: In standard offline RL problem, the static offline dataset D consists of samples {(s,a, r, s′) ∼ dD(s)πb(a|s)r(s,a)T (s′|s,a)}. Although offline RL methods learn policy for the target MDP
1For parametric Q-function, we often performQk+1 ← arg minQ E(s,a)∼D[(Bπ k+1 D Q k(s,a)−Q(s,a))2].
M := (S,A, r, T , ρ0, γ) without (costly) online data, as we shown in Figure 1, it requires a fair amount of (target) offline data D collected on M . Suppose we have another (source) offline dataset D′, consisting of samples {(s,a, r, s′) ∼ dD′(s)πb′(a|s)r(s,a)T ′(s′|s,a)} collected by the behavior policy πb′ on MDP M ′ := (S,A, r, T ′, ρ0, γ), then we hope the transfer of knowledge between offline dataset {D′∪D} can reduce the data requirements onD for learning policy for the target M .

heading: 4.1 DYNAMICS SHIFT IN OFFLINE RL
text: Although offline RL methods in Section 3 have incorporated the state-action distribution constrained backups (policy constraints or conservative estimation), they also fail to learn an adaptive policy for the target MDP M with the mixed datasets {D′ ∪ D}, as we show in Figure 4 (Appendix). We attribute this failure to the dynamics shift (Definition 2) betweenD′ andM in this adaptation setting.
Definition 1 (Empirical MDP) An empirical MDP estimated from D is M̂ := (S,A, r, T̂ , ρ0, γ) where T̂ = maxT̂ Es,a,s′∼D[log T̂ (s ′|s,a)] and T̂ (s′|s,a) = 0 for all (s,a, s′) not in dataset D.
Definition 2 (Dynamics shift) Let M̂ := (S,A, r, T̂ , ρ0, γ) be the empirical MDP estimated from D. To evaluate a policy π for M := (S,A, r, T, ρ0, γ) with offline dataset D, we say that the dynamics shift (between D and M ) in offline RL happens if there exists at least one transition pair (s,a, s′) ∈ {(s,a, s′) : dπ
M̂ (s)π(a|s)T̂ (s′|s,a) > 0} such that T̂ (s′|s,a) 6= T (s′|s,a).
In practice, for a stochastic M and any finite offline data D collected in M , there always exists the dynamics shift. The main concern is that finite samples are always not sufficient to exactly model stochastic dynamics. Following Fujimoto et al. (2019), we thus assume both MDPs M and M ′ are deterministic, which means the empirical M̂ and M̂ ′ are both also deterministic. More importantly, such assumption enables us to explicitly characterize the dynamics shift under finite offline samples.
Lemma 1 Under deterministic transition dynamics, there is no dynamics shift between D and M . For offline RL tasks, prior methods generally apply BπDQ along with the state-action distribution correction (Equations 2 and 3), which overlooks the potential dynamics shift between the (source) offline dataset and the target MDP (e.g., D′ → M ). As a result, these methods do not scale well to the setting in which dynamics shift happens, e.g., learning an adaptive policy forM with (source)D′.

heading: 4.2 DYNAMICS SHIFT IN MODEL-FREE AND MODEL-BASED OFFLINE FORMULATIONS
text: From the model-free (policy iteration) view, an exact policy evaluation on M is characterized by iteratingQ(s,a) = BπMQ(s,a) for all (s,a) such that dπM (s)π(a|s) > 0. Thus, to formalize the policy evaluation with offlineD orD′ (for an adaptive π on targetM ), we require that Bellman operator BπDQ(s,a) or BπD′Q(s,a) approximates the oracle BπMQ(s,a) for all (s,a) in Sπ or S′π , where Sπ and S′π denote the sets {(s,a) : dD(s)π(a|s) > 0} and {(s,a) : dD′(s)π(a|s) > 0} respectively. 1) To evaluate a policy π for M with D (i.e., calling the Bellman operator BπD), notable modelfree offline method BCQ (Fujimoto et al., 2019) translates the requirement of BπD = BπM into the requirement of T̂ (s′|s,a) = T (s′|s,a). Note that under deterministic environments, we have the property that for all (s,a, s′) in offline data D, T̂ (s′|s,a) = T (s′|s,a) (Lemma 1). As a result, such property permits BCQ to evaluate a policy π by calling BπD, replacing the oracle BπM , meanwhile constraining Sπ to be a subset of the support of dD(s)πb(a|s). This means a policy π which only traverses transitions contained in (target) offline data D, can be evaluated on D without error. 2) To evaluate a policy π for M with D′ (i.e., calling the Bellman operator BπD′ ), we have lemma 2: Lemma 2 Dynamics shift produces that BπD′Q(s,a) 6= BπMQ(s,a) for some (s,a) in S′π . With the offline data D′, lemma 2 suggests that the above requirement BπD′ = BπM becomes infeasible, which limits the practical applicability of prior offline RL methods under the dynamics shift. To be specific, characterizing an adaptive policy for target MDPM withD′ moves beyond the reach of the off-policy evaluation based on iterating Q = BπD′Q (Equations 2 and 3). Such iteration may cause the evaluated Q (or learned policy π) overfits to T̂ ′ and struggle to adapt to the target T . To overcome the dynamics shift, we would like to resort an additional compensation ∆T̂ ′,T such that
BπD′Q(s,a) + ∆T̂ ′,T (s,a) = B π MQ(s,a) (4)
for all (s,a) in S′π . Thus, we can apply BπD′Q+ ∆T̂ ′,T to act as a substitute for the oracle BπMQ.
From the model-based view, the oracle ηM (π) (calling the Bellman operator BπM on the target M ) and the viable ηM̂ ′(π) (calling BπM̂ ′ on the estimated M̂ ′ from sourceD′) have the following lemma.
Lemma 3 Let BπMV (s) = Ea∼π(a|s) [ r(s,a) + γEs′∼T (s′|s,a) [V (s′)] ] . For any π, we have:
ηM̂ ′(π) = ηM (π) + Es∼dπM̂′ (s) [ Bπ M̂ ′ VM (s)− BπMVM (s) ] .
Lemma 3 states that if we maximize ηM̂ ′(π) subject to |Es∼dπM̂′ (s)[B π M̂ ′ VM (s) − BπMVM (s)]| ≤ ,
ηM (π) will be improved. If F is a set of functions f : S → R that contains VM , then we have∣∣∣Es∼dπ M̂′ (s) [ Bπ M̂ ′ VM (s)− BπMVM (s) ]∣∣∣ ≤ γEs,a∼dπ M̂′ (s)π(a|s) [ dF (T̂ ′(s′|s,a), T (s′|s,a)) ] , (5) where dF (T̂ ′(s′|s,a), T (s′|s,a)) = supf∈F |Es′∼T̂ ′(s′|s,a) [f(s′)]−Es′∼T (s′|s,a) [f(s′)] |, which is the integral probability metric (IPM). Note that if we directly follow the admissible error assumption in MOPO (Yu et al., 2020) i.e., assuming dF (T̂ ′(s′|s,a), T (s′|s,a)) ≤ u(s,a) for all (s,a), this would be too restrictive: given that T̂ ′ is estimated from the source offline samples collected under T ′, not the target T , thus such error would not decrease as the source data increases. Further, we find
dF (T̂ ′(s′|s,a), T (s′|s,a)) ≤ dF (T̂ ′(s′|s,a), T̂ (s′|s,a)) + dF (T̂ (s′|s,a), T (s′|s,a)). (6)
Thus, we can bound the dF (T̂ ′, T ) term with the admissible error assumption over dF (T̂ , T ), as in MOPO, and the auxiliary constraints dF (T̂ ′, T̂ ). See next section for the detailed implementation.
In summary, we show that both prior offline model-free and model-based formulations suffer from the dynamics shift, which also suggests us to learn a modification (∆ or dF ) to eliminate this shift.

heading: 5 DYNAMICS-AWARE REWARD AUGMENTATION
text: In this section, we propose the dynamics-aware reward augmentation (DARA), a simple data augmentation procedure based on prior (model-free and model-based) offline RL methods. We first provide an overview of our offline reward augmentation motivated by the compensation ∆T̂ ′,T in Equation 4 and the auxiliary constraints dF (T̂ ′, T̂ ) in Equation 6, and then describe its theoretical derivation in both model-free and model-based formulations. With the (reduced) target offline data D and the source offline data D′, we summarize the overall DARA framework in Algorithm 1.
Algorithm 1 Framework for Dynamics-Aware Reward Augmentation (DARA) Require: Target offline data D (reduced) and source offline data D′
1: Learn classifiers (qsas and qsa) that distinguish source data D′ from target data D. (See Appendix A.1.3) 2: Set dynamics-aware ∆r(st,at, st+1) = log
qsas(source|st,at,st+1) qsas(target|st,at,st+1) − log qsa(source|st,at) qsa(target|st,at) .
3: Modify rewards for all (st,at, rt, st+1) in D′: rt ← rt − η∆r. 4: Learn policy with {D ∪ D′} using prior model-free or model-based offline RL algorithms.

heading: 5.1 DYNAMICS-AWARE REWARD AUGMENTATION IN MODEL-FREE FORMULATION
text: Motivated by the well established connection of RL and probabilistic inference (Levine, 2018), we first cast the model-free RL problem as that of inference in a particular probabilistic model. Specifically, we introduce the binary random variable O that denotes whether the trajectory τ := (s0,a0, s1, ...) is optimal (O = 1) or not (O = 0). The likelihood of a trajectory can then be modeled as p(O = 1|τ) = exp ( ∑ t rt/η), where rt := r(st,at) and η > 0 is a temperature parameter.
(Reward Augmentation with Explicit Policy/Value Constraints) We now introduce a variational distribution pπ
M̂ ′ (τ) = p(s0) ∏ t=1 T̂
′(st+1|st,at)π(at|st) to approximate the posterior distribution pπM (τ |O = 1), which leads to the evidence lower bound of log pπM (O = 1):
log pπM (O = 1) = logEτ∼pπM (τ) [p(O = 1|τ)] ≥ Eτ∼pπM̂′ (τ)
[ log p(O = 1|τ) + log p π M (τ)
pπ M̂ ′ (τ)
]
= Eτ∼pπ M̂′ (τ) [∑ t ( rt/η − log T̂ ′(st+1|st,at) T (st+1|st,at) )] . (7)
Since we are interested in infinite horizon problems, we introduce the discount factor γ and take the limit of steps in each rollout, i.e., H → ∞. Thus, the RL problem on the MDP M , cast as the inference problem arg maxπ log p π M (O = 1), can be stated as a maximum of the lower bound
Eτ∼pπ M̂′ (τ)
[∑∞ t=0 γ t ( rt − η log T̂ ′(st+1|st,at) T (st+1|st,at) )] . This is equivalent to an RL problem on M̂ ′ with
the augmented reward r ← r(s,a)− η log T̂ ′(s′|s,a) T (s′|s,a) . Intuitively, the −η log T̂ ′(s′|s,a) T (s′|s,a) term discourages transitions (state-action-next-state) in D′ that have low transition probability in the target M . In the model-free offline setting, we can add the explicit policy or Q-value constraints (Equations2 and 3) to mitigate the OOD state-actions. Thus, such formulation allows the oracle BπM to be reexpressed by BπD′ and the modification log T̂ ′ T , which makes the motivation in Equation 4 practical.
(Reward Augmentation with Implicit Policy Constraints) If we introduce the variational distribution pπ ′
M̂ ′ (τ) := p(s0) ∏ t=1 T̂
′(st+1|st,at)π′(at|st), we can recover the weighted-regressionstyle (Wang et al., 2020; Peng et al., 2019; Abdolmaleki et al., 2018; Peters et al., 2010) objective by maximizing J (π′, π) := Eτ∼pπ′
M̂′ (τ)
[∑∞ t=0 γ t ( rt − η log T̂ ′(st+1|st,at) T (st+1|st,at) − η log π′(at|st) π(at|st) )] (lower
bound of log pπM (O = 1)). Following the Expectation Maximization (EM) algorithm, we can maximize J (π′, π) by iteratively (E-step) improving J (π′, ·) w.r.t. π′ and (M-step) updating π w.r.t. π′.
(E-step) We define Q̃(s,a, s′) = Eτ∼pπ′ M̂′ (τ)
[∑ t γ t log T̂ ′(s′|s,a) T (s′|s,a) |s0 = s,a0 = a, s1 = s ′ ] . Then,
given offline data D′, we can rewrite J (π′, ·) as a constrained objective (Abdolmaleki et al., 2018):
max π′ Es,a,s′∼dD′ (s)π′(a|s)T̂ ′(s′|s,a) [ Q(s,a)− ηQ̃(s,a, s′) ] , s.t. Es∼dD′ (s) [DKL (π ′(a|s)‖π(a|s))] ≤ .
When considering a fixed π, the above optimization over π′ can be solved analytically (Vieillard et al., 2020; Geist et al., 2019; Peng et al., 2019). The optimal π′∗ is then given by π ′ ∗(a|s) ∝ π(a|s) exp (Q(s,a)) exp(−ηQ̃(s,a, T̂ ′(s′|s,a))). As the policy evaluation in Equation 1 (Footnote2), we estimate Q(s,a) and Q̃(s,a, s′) by minimizing the Bellman error with offline samples in D′. (M-step) Then, we can project π′∗ onto the manifold of the parameterized π:
arg min π
Es∼dD′ (s) [DKL (π ′ ∗(a|s)‖π(a|s))]
= arg max π
Es,a,s′∼D′ [ log π(a|s) exp (Q(s,a)) exp ( −ηQ̃(s,a, s′) )] . (8)
From the regression view, prior work MPO (Abdolmaleki et al., 2018) infers actions with Q-value weighted regression, progressive approach compared to behavior cloning; however, such paradigm lacks the ability to capture transition dynamics. We explicitly introduce the exp(−ηQ̃(s,a, s′)) term, which as we show in experiments, is a crucial component for eliminating the dynamics shift.
Implementation: In practice, we adopt offline samples in D to approximate the true dynamics T of M , and introduce a pair of binary classifiers, qsas(·|s,a, s′) and qsa(·|s,a), to replace log T̂ ′(s′|s,a) T (s′|s,a) as in Eysenbach et al. (2021): log T̂ ′(s′|s,a) T (s′|s,a) = log qsas(source|s,a,s′) qsas(target|s,a,s′) − log qsa(source|s,a) qsa(target|s,a) . (See AppendixA.1.3 for details). Although the amount of dataD sampled from the targetM is reduced in our problem setup, we experimentally find that such classifiers are sufficient to achieve good performance.

heading: 5.2 DYNAMICS-AWARE REWARD AUGMENTATION IN MODEL-BASED FORMULATION
text: Following Equation 6, we then characterize the dynamics shift compensation term as in the above model-free analysis in the model-based offline formulation. We will find that across different derivations, our reward augmentation ∆r has always maintained the functional consistency and simplicity.
Following MOPO, we assume F = {f : ‖f‖∞ ≤ 1}, then we have dF (T̂ ′(s′|s,a), T̂ (s′|s,a)) = DTV(T̂ ′(s′|s,a), T̂ (s′|s,a)) ≤ (DKL(T̂ ′(s′|s,a), T̂ (s′|s,a))/2) 1 2 , where DTV is the total variance distance. Then we introduce the admissible error u(s,a) such that dF (T̂ (s′|s,a), T (s′|s,a)) ≤ u(s,a) for all (s,a), and η and δ such that (DKL(T̂ ′, T̂ )/2) 1 2 ≤ ηDKL(T̂ ′, T̂ ) + δ. Following Lemma 3, we thus can maximize the following lower bound with the samples in M̂ ′ (λ := γRmax1−γ ):
ηM (π) ≥ Es,a,s′∼dπ M̂′ (s)π(a|s)T̂ ′(s′|s,a)
[ r(s,a)− ηλ log T̂
′(s′|s,a) T̂ (s′|s,a) − λu(s,a)− λδ
] . (9)
Implementation: We model the dynamics T̂ ′ and T̂ with an ensemble of 2*N parameterized Gaussian distributions: N i
T̂ ′ (µθ′(s,a),Σφ′(s,a)) and N iT̂ (µθ(s,a),Σφ(s,a)), where i ∈ [1, N ]. We
approximate u with the maximum standard deviation of the learned models in the ensemble: u(s,a) = maxNi=1 ‖Σφ(s,a)‖F, omit the training-independent δ, and treat λ as a hyperparameter as in MOPO. For the log T̂ ′
T̂ term, we resort to the above classifiers (qsas and qsa) in model-free set-
ting. (See Appendix-A.3.2 for comparison between using classifiers and estimated-dynamics ratio.)

heading: 5.3 REGULARIZATION VIEW OF DYNAMICS-AWARE REWARD AUGMENTATION
text: Here we shortly characterize our dynamics-aware reward augmentation from the density regularization. Note the standard max-return objective ηM (π) in RL can be written exclusively in terms of the on-policy distribution dπM (s)π(a|s). To introduce an off-policy distribution dD(s)πb(a|s) in the objective, prior works often incorporate a regularization (penalty): D(dπM (s)π(a|s)‖dD(s)πb(a|s)), as in Equations 2 and 3. However, facing dynamics shift, such regularization should take into account the transition dynamics, which is penalizing D(dπM (s)π(a|s)T (s′|s,a)‖dD′(s)πb′(a|s)T̂ ′(s′|s,a)). From this view, it is also straightforward to derive the reward modification built on prior regularized off-policy max-return objective2 e.g., the off-policy approach AlgaeDICE (Nachum et al., 2019b).

heading: 6 EXPERIMENTS
text: We present empirical demonstrations of our dynamics-aware reward augmentation (DARA) in a variety of settings. We start with two simple control experiments that illustrate the significance of DARA under the domain (dynamics) adaptation setting. Then we incorporate DARA into state-ofthe-art (model-free and model-based) offline RL methods and evaluate the performance on the D4RL tasks. Finally, we compare our framework to several cross-domain-based baselines on simulated and real-world tasks. Note that for the dynamics adaptation, we also release a (source) dataset as a complement to D4RL, along with the quadruped robot dataset in simulator (source) and real (target).

heading: 6.1 HOW DOES DARA HANDLE THE DYNAMICS SHIFT IN OFFLINE SETTING?
text: source/target
goal ×× w /o
A ug
. source target
D A
R A
Q (w/o Aug.) Q (DARA)
40 20
0
(s1, a1) (s1, a2)
Here we characterize both external and internal dynamics shifts: In Map tasks (Figure 2 left), the source dataset D′ is collected in a 2D map and the target D is collected in the same environment but with an obstacle (the dashed line); In Ant tasks (Figure 3 left), the source datasetD′ is collected using the Mujoco Ant and the target D is collected with the same Ant but one joint of which is restricted.
2We list some related works in Appendix A.2, where the majority of the existing work focuses on regularizing state-action distribution, while dynamics shift receives relatively little attention. Thus, we hope to shift the focus of the community towards analyzing how the dynamics shift affects RL and how to eliminate the effect.
Using MPO, as an example of offline RL method, we train a policy on dataset {D′ ∪D} and deploy the acquired policy in both source and target MDPs. As shown in Figure 2 (middle-top, w/o Aug.), such training paradigm does not produce an adaptive policy for the target. By modifying rewards in sourceD′, we show that applying the same training paradigm on the reward augmented data exhibits a positive transfer ability in Figure 2 (middle-bottom, DARA). In Figure 2 (right), we show that our DARA produces low Q-values on the obstructive state-action pairs (in left) compared to the vanilla MPO, which thus prevents the Q-value weighted-regression on these unproductive state-action pairs.
More generally, we illustrate how DARA can handle the dynamics adaptation from the reward modification view. In Figure 3 (right), the learned reward modification −∆r (dashed blue line) clearly produces a penalty (red slices) on these state-action pairs (in source) that produce infeasible nextstate transitions in the target MDP. If we directly apply prior offline RL methods, these transitions that are beyond reach in target and yet are high valued, would yield a negative transfer. Thus, we can think of DARA as finding out these transitions that exhibit dynamics shifts and enabling dynamics adaptation with reward modifications, e.g., penalizing transitions covered by red slices (−∆r < 0).

heading: 6.2 CAN DARA ENABLE AN ADAPTIVE POLICY WITH REDUCED OFFLINE DATA IN TARGET?
text: To characterize the offline dynamics shift, we consider the Hopper, Walker2d and Halfcheetah from the Gym-MuJoCo environment, using offline samples from D4RL as our target offline dataset. For the source dataset, we change the body mass of agents or add joint noise to the motion, and, similar to D4RL, collect the Random, Medium, Medium-R and Medium-E offline datasets for the three environments. Based on various offline RL algorithms (BEAR, BRAC-p, BCQ, CQL, AWR, MOPO), we perform the following comparisons: 1) employing the 100% of D4RL data (10T), 2) employing only 10% of the D4RL data (1T), 3) employing 10% of the D4RL data and 100% of our collected source offline data (1T+10S w/o Aug.), and 4) employing 10% of the D4RL data and 100% of our collected source offline data along with our reward augmentation (1T+10S DARA). Due to page limit, here we focus on the dynamics shift concerning the body mass on Walker2d and Hopper. We refer the reader to appendix for more experimental details, tasks, and more baselines (BC, COMBO).
As shown in Table 1, in most of the tasks, the performance degrades substantially when we decrease the amount of target offline data, i.e., 10T → 1T. Training with additional ten times source offline data (1T+10S w/o Aug.) also does not bring substantial improvement (compensating for the reduced data in target), which even degrades the performance in some tasks. We believe that such degradation (compared to 10T) is caused by the lack of target offline data as well as the dynamics shift (induced
by the source data). Incorporating our reward augmentation, we observe that compared to 1T and 1T+10S w/o Aug. that both use 10% of the target offline data, our 1T+10S DARA significantly improves the performance across a majority of tasks. Moreover, DARA can achieve comparable or better performance compared to baseline 10T that training with ten times as much target offline data.

heading: 6.3 CAN DARA PERFORM BETTER THAN CROSS-DOMAIN BASELINES?
text: In Section 6.2, 1T+10S w/o Aug. does not explicitly learn policy for the target dynamics, thus one proposal (1T+10S fine-tune) for adapting the target dynamics is fine-tuning the model that learned with source offline data, using the (reduced) target offline data. Moreover, we also compare DARA with the recently proposed MABE (Cang et al., 2021), which is suitable well for our cross-dynamics setting by introducing behavioral priors πp in the model-based offline setting. Thus, we implement two baselines: 1) 1T+10S MABE πpT̂ and 2) 1T+10S MABE T̂ πp, which denotes 1) learning πp with target domain data and T̂ with source domain data, and 2) learning πp with source domain data and T̂ with target domain data, respectively. We show the results for the Walker (with body mass shift) in Table 2, and more experiments in Appendix A.3.5. Our results show that DARA achieves significantly better performance than the naı̈ve fine-tune-based approaches in a majority of tasks (67 ”↑” vs. 13 ”↓”, including results in appendix). On twelve out of the sixteen tasks (including results in appendix), DARA-based methods outperform the MABE-based methods. We attribute MABE’s failure to the difficulty of the reduced target offline data, which limits the generalization of the learned πp or T̂ under such data. However, such reduced data (10% of target) is sufficient to modify rewards in the source offline data, which thus encourages better performance for our DARA.
For real-world tasks, we also test DARA in a new offline dataset on the quadruped robot (see appendix for details). Note that we can not access the privileged information (e.g., coordinate) in real robot, thus the target offline data (collected in real-world) does not contain rewards. This means that prior fine-tune-based and MABE-based methods become unavailable. However, our reward augmentation frees us from the requisite of rewards in
target domain. We can freely perform offline training only using the augmented source offline data as long as the learned ∆r is sufficient. For comparison, we also employ a baseline (w/o Aug.): directly deploying the learned policy with source data into the (target) real-world. We present the results (deployed in real with obstructive stairs) in Table 3 and videos in supplementary material. We can observe that training with our reward augmentation, this performance can be substantially improved. Due to page limit, we refer readers to Appendix A.3.6 for more experimental results and discussion.

heading: 7 CONCLUSION
text: In this paper, we formulate the dynamics shift in offline RL. Based on prior model-based and modelfree offline algorithms, we propose the dynamics-award reward augmentation (DARA) framework that characterizes constraints over state-action-next-state distributions. Empirically we demonstrate DARA can eliminate the dynamics shift and outperform baselines in simulated and real-world tasks.

heading: A APPENDIX

heading: A.1 DERIVATION

heading: A.1.1 PROOF OF LEMMA 3
text: Let BπMV (s) = Ea∼π(a|s) [ r(s,a) + γEs′∼T (s′|s,a) [V (s′)] ] and r(s) = Ea∼π(a|s) [r(s,a)]. Then, we have ηM̂ ′(π)− ηM (π) = Es0∼ρ0(s) [ VM̂ ′(s0)− VM (s0)
] =
∞∑ t=0 γtEst∼P (st|π,M̂ ′,t)Eat∼π(at|st) [r(st,at)]− Es0∼ρ0(s) [VM (s0)]
= ∞∑ t=0 γtEst∼P (st|π,M̂ ′,t) [r(st) + VM (st)− VM (st)]− Es0∼ρ0(s) [VM (s0)]
= ∞∑ t=0 γtEst∼P (st|π,M̂ ′,t) st+1∼P (st+1|π,M̂ ′,t+1) [r(st) + γVM (st+1)− VM (st)]
= ∞∑ t=0 γtEst∼P (st|π,M̂ ′,t) st+1∼P (st+1|π,M̂ ′,t+1) [ r(st) + γVM (st+1)− ( r(st) + γEa∼π(a|st),s′∼T (st,a) [VM (s ′)] )]
= ∞∑ t=0 γtEst∼P (st|π,M̂ ′,t) [ Bπ M̂ ′ VM (st)− BπMVM (st) ] = Es∼dπ
M̂′ (s) [ Bπ M̂ ′ VM (s)− BπMVM (s) ] .

heading: A.1.2 MODEL-BASED FORMULATION
text: Here we provide detailed derivation of the lower bound in Equation 9 in the main text.
Assumption 1 Assume a scale c and a function class F such that VM ∈ cF .
Following MOPO (Yu et al., 2020), we set F = {f : ‖f‖∞ ≤ 1}. In Section Preliminaries, we have that the reward function is bounded: r(s,a) ∈ [−Rmax, Rmax]. Thus, we have ‖VM‖∞ ≤∑∞ t=0 γ tRmax = Rmax 1−γ and hence the scale c = Rmax 1−γ . As a direct corollary of Assumption 1 and Equation 5, we have∣∣∣Es∼dπ M̂′ (s) [ Bπ M̂ ′ VM (s)− BπMVM (s) ]∣∣∣ ≤ γc · Es,a∼dπ M̂′ (s)π(a|s) [ dF (T̂ ′(s′|s,a), T (s′|s,a)) ] .
(10)
Further, we find dF (T̂ ′(s′|s,a), T (s′|s,a)) ≤ dF (T̂ ′(s′|s,a), T̂ (s′|s,a)) + dF (T̂ (s′|s,a), T (s′|s,a)) (11)
For the first term dF (T̂ ′(s′|s,a), T̂ (s′|s,a)) in Equation 12, through Pinsker’s inequality, we have
dF (T̂ ′(s′|s,a), T̂ (s′|s,a)) = DTV(T̂ ′(s′|s,a), T̂ (s′|s,a)) ≤
√ 1
2 DKL(T̂ ′(s′|s,a), T̂ (s′|s,a))
(12) To keep consistent with the DARA-based method-free offline methods, we introduce scale η and bias δ to eliminate the square root in Equation 12. To be specific, we assume3 scale η and bias δ
such that √
1 2DKL(T̂ ′, T̂ ) ≤ ηDKL(T̂ ′, T̂ ) + δ. Thus, we obtain
dF (T̂ ′(s′|s,a), T̂ (s′|s,a)) = DTV(T̂ ′(s′|s,a), T̂ (s′|s,a)) ≤ ηDKL(T̂ ′(s′|s,a), T̂ (s′|s,a)) + δ
(13)
3In implementation, we clip the maximum deviation of log T̂ ′(s′|s,a) T̂ (s′|s,a) for each (s,a, s ′), which thus makes
DKL(T̂ ′(s′|s,a), T̂ (s′|s,a)) bounded.
For the second term dF (T̂ (s′|s,a), T (s′|s,a)) in Equation 11, we assume that we have access to an oracle uncertainty qualification module that provides an upper bound on the error of the estimated empirical MDP M̂ := {S,A, r, T̂ , ρ0, γ}.
Assumption 2 LetF be the function class in Assumption 1. We say u : S×A → R is an admissible error estimator for T̂ if dF (T̂ (s′|s,a), T (s′|s,a)) ≤ u(s,a) for all (s,a).
Thus, we have
Es,a∼dπ M̂′ (s)π(a|s)
[ dF (T̂ (s ′|s,a), T (s′|s,a)) ] ≤ Es,a∼dπ
M̂′ (s)π(a|s) [u(s,a)] (14)
Bring Inequations 10, 11, 13, and 14 into Lemma 3, we thus have
ηM (π) ≥ Es,a,s′∼dπ M̂′ (s)π(a|s)T̂ ′(s′|s,a)
[ r(s,a)− ηγc log T̂
′(s′|s,a) T̂ (s′|s,a) − γcu(s,a)− γcδ
] . (15)

heading: A.1.3 LEARNING CLASSIFIERS
text: Applying Bayes’ rule, we have
T̂ ′(s′|a, s) := p(s′|s,a, source) = p(source|s,a, s ′)p(s,a, s′)
p(source|s,a)p(s,a) ,
T̂ (s′|a, s) := p(s′|s,a, target) = p(target|s,a, s ′)p(s,a, s′)
p(target|s,a)p(s,a) .
Then we parameterize p(·|s,a, s′) and p(·|s,a) with the two classifiers qsas and qsa respectively. Using the standard cross-entropy loss, we learn qsas and qsa with the following optimization objective:
max E(s,a,s′)∼D′ [log qsas(source|s,a, s′)] + E(s,a,s′)∼D [log qsas(target|s,a, s′)] , max E(s,a)∼D′ [log qsa(source|s,a)] + E(s,a)∼D [log qsa(target|s,a)] .
With the trained qsas and qsa, we have
log T̂ ′(s′|s,a) T̂ (s′|s,a) = log qsas(source|s,a, s′) qsas(target|s,a, s′) − log qsa(source|s,a) qsa(target|s,a) . (16)
In our implementation, we also clip the above reward modification between −10 and 10.

heading: A.2 REGULARIZATION VIEW
text: In Table 4, we provide some related works with respect to the (state-action pair) dD(s)πb(a|s) regularization and the (state-action-next-state pair) dD′(s)πb′(a|s)T̂ ′(s′|s,a) regularization.

heading: A.3 MORE EXPERIMENTS

heading: A.3.1 TRAINING WITH {D′ ∪ D}
text: As we show in Figure 1 in Section Introduction, the performance of prior offline RL methods deteriorates dramatically as the amount of (target) offline data D decreases. In Figure 4, we show that directly training with the mixed dataset {D′ ∪ D} will not compensate for the deteriorated performance caused by the reduced target offline data, and training with such additional source offline data can even lead the performance degradation in some tasks.

heading: A.3.2 COMPARISON BETWEEN LEARNING CLASSIFIERS AND LEARNING DYNAMICS (FOR THE REWARD MODIFICATION)
text: In Table 5, we show the comparison between learning classifiers and learning dynamics (for our reward modification) in the Hopper tasks. We can observe that the two schemes for estimating the reward modification have similar performance. Thus, for simplicity and following Eysenbach et al. (2021), we adopt the classifiers to modify rewards in the source offline data in our experiments.

heading: A.3.3 MORE EXAMPLES WITH RESPECT TO THE REWARD AUGMENTATION
text: In Figure 5, we provide more examples with respect to the reward augmentation in the Ant task in Figure 3 (left).
A.3.4 COMPARISON BETWEEN 10T, 1T, 1T+10S w/o Aug., AND 1T+10S DARA
Based on various offline RL algorithms (BEAR (Kumar et al., 2019), BRAC-p (Wu et al., 2019), BCQ (Fujimoto et al., 2019), CQL (Kumar et al., 2020a), AWR (Peng et al., 2019), MOPO (Yu et al., 2020), BC (behavior cloning), COMBO (Yu et al., 2021)), we provide the additional results in Tables 6, 7, 8, 9, and 10.

heading: A.3.5 COMPARISON WITH THE CROSS-DOMAIN BASED BASELINES
text: In Tables 11 and 12, we provide the comparison between our DARA-based methods, fine-tune based methods, and MABE-based methods in Hopper and Walker2d tasks, over the dynamics shift concerning the joint noise of motion. We can observe that in a majority of tasks, our DARAbased methods outperforms the fine-tune-based method (67 ”↑” vs. 13 ”↓”, including the results in the main text). Moreover, our DARA can achieve comparable or better performance compared to MABE-based baselines on eleven out of sixteen tasks (including the results in the main text).

heading: A.3.6 ADDITIONAL RESULTS ON THE QUADRUPED ROBOT
text: In this offline sim2real setting, we collect the source offline data in the simulator (106 or 2 ∗ 106 steps) and target offline data in the real world (3 ∗ 104 steps). See Appendix A.4 for details. For testing, we directly deploy the learned policy in the real (flat or obstructive) environment and adopt the average distance covered in an episode (300 steps) as our evaluation metrics.
(Flat and static environment) We first deploy our learned policy in the flat and static environment. The results (distance covered in an episode) are provided in Table 13.
1) BCQ (Figure 7): We find that with Medium-R offline data, w/o Aug. BCQ and DARA BCQ both could not acquire the locomotion skills, which we think is caused by the lack of high-quality offline data. With more ”expert” data (Medium-R → Medium → Medium-E, or Medium-R → MediumR-E), w/o-Aug. BCQ allows for progressive performance (0.00→ 1.56→ 2.16, or 0.00→ 1.69 in BCQ), but with our reward augmentation, such performance can be further improved (with average improvement 13.6%).
2) CQL (Figure 8): We find that with Medium-R or Medium-R-E offline data, w/o Aug. CQL and DARA CQL both could not learn the locomotion skills, which we think is caused by the low-quality ”Replay” offline data. With Medium or Medium-E offline data, w/o Aug. CQL and DARA CQL acquire similar performance on this flat and static environment.
3) MOPO: We find that the model-based MOPO (both w/o Aug. and DARA) could hardly learn the locomotion skill under the provided offline data.
(Obstructive and dynamic environment) We then deploy our learned policy in the obstructive and dynamic environment. The results (distance covered in an episode) are provided in Table 14.
1) BCQ (Figure 9): In this obstructive environment, we can obtain similar results as in the flat environment. With more ”expert” data (Medium → Medium-E → Medium-R-E), w/o Aug. BCQ allows for progressive performance (0.85 → 1.15 → 1.27), and with our reward augmentation, such performance can be further improved (with average improvement 25.9%). At the same time, we can also find that due to the presence of environmental obstacles, the performance of both w/o
Aug. BCQ and w/o Aug. DARA is decreased compared to the deployment on the flat environment. However, we find that our DARA exhibits greater average performance improvement under this obstructive environment (13.6% → 25.9%) compared to that in the flat environment. These results demonstrate that our DARA can learn an adaptive policy for the target environment and thus show a greater advantage over w/o-Aug. in more complex environments.
2) CQL (Figure 10): Similar to BCQ, our DARA CQL exhibits a greater performance improvement over baseline in the obstructive and dynamic environment (0.2%→ 30.9%) compared to that in the flat and static environment.
In summary, The results in the quadruped robot tasks support our conclusion in the main text regarding the dynamics shift problem in offline RL — with only modest amounts of target offline data (3 ∗ 104 steps), DARA-based methods can acquire an adaptive policy for the (both flat and obstructive) target environment and exhibit better performance compared to baselines under the dynamics adaptation setting.

heading: A.3.7 ABLATION STUDY WITH RESPECT TO THE AMOUNT OF TARGET OFFLINE DATA
text: To see whether the amount of target offline data can be further reduced, we show the results of the ablation study with respect to the amount of target offline data in Tables 15 and 16.

heading: A.4 ENVIRONMENTS AND DATASET
text: In this work, the tasks include Hopper, Walker2d, HalfCheetah, simulated and real-world quadruped robot, which are illustrated in Figure 11.
In the Hopper, Walker2d and HalfCheetah dynamics adaptation setting, we set the D4RL (Fu et al., 2020) dataset as our target domain. For the source dynamics, we change the body mass (body mass shift) or add noises to joints (joint noise shift) of the agents (see Table 17 for the details) and then collect the source offline dataset in the changed environment. Following Fu et al. (2020), on the changed source environment, we collect the 1) ”Random” offline data, generated by unrolling a randomly initialized policy, 2) ”Medium” offline data, generated by a trained policy with the “medium” level of performance in the source environment, 3) ”Medium-Replay” (Medium-R) offline data, consisting of recording all samples in the replay buffer observed during training until the policy reaches the “medium” level of performance, 4) ”Medium-Expert” (Medium-E) offline data, mixing equal amounts of expert demonstrations and ”medium” data in the source environment.
In the sim2real setting (for the quadruped robot), we use the A1 dog from Unitree (Wang, 2020). We collect the target offline data using five target behavior policies in the real-world with changing terrains, as shown in Figure 12, and collect the ”Medium”, ”Medium-Replay” (Medium-R), ”MediumExpert” (Medium-E), ”Medium-Replay-Expert” (Medium-RE) source offline data in the simulator, where ”MediumReplay-Expert” denotes mixing equal amounts of ”MediumReplay” data and expert demonstrations in the simulator. In Section A.5, we provide the details of how to obtain the target and source behavior policy, so as to collect our target and source offline data.
We list our tasks properties in Table 18 and provide our collected dataset in supplementary material. In implementation, we set η = 0.1 for all simulated tasks and set η = 0.01 for the sim2real task. In Table 23, we also report the sensitivity of DARA on the hyper-parameters η.

heading: A.5 TRAINING THE (TARGET AND SOURCE) BEHAVIOR POLICY FOR THE QUADRUPED ROBOT
text: To obtain a behavior policy that can be deployed in simulator (for collecting the source offline data) or real-world (for collecting the target offline data), we introduce the prior knowledge (Iscen et al., 2018) and domain randomization (Tobin et al., 2017; Peng et al., 2018).
Prior Knowledge: To reduce the impact of the foot at the moment of touching the ground during the robot locomotion, we designed a compound cycloid trajectory (Sakakibara et al., 1990) as prior knowledge. In our implementation for the foot trajectory, four aspects are mainly considered: 1) The robot walks stably without obvious shaking; 2) The joint impact of the robot during the locomotion is small; 3) The joint speed and acceleration of the robot during the locomotion are continuous and smooth; 4) The feet of the robot will not slide when they are in contact with the ground. Similar to Lee et al. (2020), we define a periodic phase variable φi ∈ [0.0, 0.6), i = 1, 2, 3, 4 for each leg, which represents swing phase if φi ∈ [0.0, 0.3) and contact phase if φi ∈ [0.3, 0.6). At every time step t, φi = (t ∗ f0 + φ0[i] + φoffset[i])(mod 2Tm) where Tm = 0.3, and f0 = 1.1 is the base frequency, and φ0 = [0, 0.3, 0.3, 0] is the initial phase. φoffset is part of the output of the controller. The trajectory of the swing leg is: xi = S [ t Tm − 12π sin ( 2πt Tm )] + S0, i = 1, 2 xi = S [ t Tm − 12π sin ( 2πt Tm )] − S + S0, i = 3, 4
y = Y0 z = H [ sgn ( Tm 2 − t ) (2fE(t)− 1) + 1 ] + Z0
,
where
fE(t) = t Tm − 1 4π sin
( 4πt
Tm
) ,
and
sgn ( Tm 2 − t ) = { 1 0 ≤ t < Tm2 −1 Tm2 ≤ t < Tm .
The trajectory of the standing leg is: xi = S ( 2Tm−t Tm + 12π sin ( 2πt Tm )) + S0, i = 1, 2 xi = S ( 2Tm−t Tm + 12π sin ( 2πt Tm )) − S + S0, i = 3, 4
y = Y0 z = Z0
.
where S = 0.14m,H = 0.18m are the maximum foot length and height. S0 = [0.17, 0.17,−0.2,−0.2], Y0 = [−0.13, 0.13,−0.13, 0.13], Z0 = [−0.32,−0.32,−0.32,−0.32] are the default target foot position in body frame.
Domain Randomization: To encourage the policy to be robust to variations in the dynamics, we incorporate the domain randomization. In Table 19, we provide the dynamics parameters and their respective range of values.
State Space, Action Space and Reward Function: The action is a 16-dimensional vector consisting of leg phase and target foot position residuals in the body frame. The design of state space and reward function mainly follows the prior work Lee et al. (2020). In Table 20, we provide the state representation.
The reward function is defined as
0.1rlv + 0.05ry + 0.05rrp + 0.005rb + 0.02rbc + 0.025rs + 2 · 10−5rτ .
The individual terms are defined as follows.
1) Linear velocity reward rlv :
rlv := { exp(−30|vpr − 0.2|) vpr < 0.2 1 vpr ≥ 0.2 ,
where vpr = vxy · v̂xy is the base linear velocity projected onto the command direction. 2) Yaw angle reward ry : ry := exp(−(y − ŷ)2), (17) where y and ŷ is the yaw and desired yaw angle.
3) Roll and pitch reward rrp : rrp := exp(−1.5 ∑ (φ− [0, arccos(< Pxz, (0, 0, 1) T >
‖ Pxz ‖ )− π/2])2), (18)
where φ are the roll and pitch angle. Pxz = P1 − P4 or Pxz = P2 − P3,Pi, i ∈ [1, 4] are the foot position in world frame. The advantage of designing the target pitch angle in this way is to ensure that the body of the robot is parallel to the supporting surface of the stand legs, thereby ensuring that the robot can smoothly over challenge terrain, such as upward stairs.
4) Base motion reward rb : rb := exp(−1.5 ∑ (vxy − vpr ∗ v̂xy)2) + exp(−1.5 ∑ (ωxy) 2), (19)
where ωxy are the roll and pitch rates.
5) Body collision reward rbc : rbc := −|Ibody/Ifoot|, (20) where Ibody and Ifoot are the contact numbers of robot’s body parts and foot with the terrain, respectively.
6) Target smooth reward rs : rs := −||fd,t − 2fd,t−1 + fd,t−2||, (21) where fd,i(i = t, t− 1, t− 2) are the target foot positions in the time-step t, t− 1 and t− 2. 7) Torqure reward rτ :
rτ := − ∑ i |τi|, (22)
where τi is the joint torques.
Training Details: Both the behavior policy and value networks are Multilayer Perceptron (MLP) with 3 hidden layers, which have 256, 128 and 64 nodes. The activation function is the Tanh function, and the optimizer is Adam. With the above prior knowledge, domain randomization and reward function, we train our behavior policy with SAC (Haarnoja et al., 2018) in PyBullet (Coumans & Bai, 2016–2021).

heading: A.6 ADDITIONAL RESULTS
text: Ta bl
e 21
:N or
m al
iz ed
sc or
es fo
rt he
D 4R
L ta
sk s
(w ith
bo dy
m as
s sh
if t)
.W e
ta ke
th e
ba se
lin e
re su
lts (f
or 10
T )o
fM O
PO fr
om th
ei ro
ri gi
na lp
ap er
s an
d th
at of
th e
ot he
rm od
el -f
re e
m et
ho ds
(B E
A R
,B R
A C
-p ,A
W R
,B C
Q an
d C
Q L
)f ro
m th
e D
4R L
pa pe
r( Fu
et al
., 20
20 ).
B od
y M
as s
Sh if
t 10
T 1T
1T +1
0S 1T
+1 0S
10 T
1T 1T
+1 0S
1T +1
0S 10
T 1T
1T +1
0S 1T
+1 0S
w /o
A ug
. (D
A R
A )
w /o
A ug
. (D
A R
A )
w /o
A ug
. (D
A R
A )
Hopper
B E
A R
B R
A C
-p A
W R
R an
do m
11 .4
1 ±
0. 5
4. 6 ±
3. 4
8. 4 ±
1. 2
11 10
.9 ±
0. 1
9. 6 ±
3. 3
11 ±
0. 6
10 .2
10 .3 ±
0. 3
3. 4 ±
0. 7
4. 5 ±
0. 9
M ed
iu m
52 .1
0. 8 ±
0 0.
9 ±
0. 3
1. 6 ±
1 32
.7 29 ±
6. 2
29 .2 ±
2. 1
32 .9 ±
7. 5
35 .9
30 .9 ±
0. 4
20 .8 ±
6. 3
28 .9 ±
5. 5
M ed
iu m
-R 33
.7 1.
3 ±
1. 5
18 .2 ±
5 34
.1 ±
5. 8
0. 6
5. 4 ±
3. 3
20 .1 ±
4. 8
30 .8 ±
4. 9
28 .4
8. 8 ±
4. 9
4. 1 ±
1. 7
4. 2 ±
3. 5
M ed
iu m
-E 96
.3 0.
8 ±
0. 1
0. 6 ±
0 1.
2 ±
0. 5
1. 9
34 .5 ±
14 .7
32 .3 ±
7. 8
34 .7 ±
8. 5
27 .1
27 ±
1. 3
26 .8 ±
0. 4
26 .6 ±
2
Hopper
B C
Q C
Q L
M O
PO
R an
do m
10 .6
10 .6 ±
0. 1
8. 3 ±
0. 3
9. 7 ±
0. 2
10 .8
10 .6 ±
0. 1
10 .2 ±
0. 3
10 .4 ±
0. 4
11 .7
4. 8 ±
2. 4
2 ±
2. 1
2. 1 ±
1. 7
M ed
iu m
54 .5
37 .1 ±
6. 3
25 .7 ±
5. 5
38 .4 ±
1. 8
58 43 ±
9. 2
44 .9 ±
2. 7
59 .3 ±
12 .2
28 4.
1 ±
2 5 ±
5. 3
10 .7 ±
5. 1
M ed
iu m
-R 33
.1 9.
3 ±
4. 4
28 .7 ±
1. 9
32 .8 ±
0. 9
48 .6
9. 6 ±
5. 2
1. 4 ±
0. 3
3. 7 ±
1. 4
67 .5
1 ±
0. 6
5. 5 ±
4. 6
8. 4 ±
3. 5
M ed
iu m
-E 11
0. 9
58 ±
16 .2
75 .4 ±
7. 8
84 .2 ±
9. 8
98 .7
59 .7 ±
34 .5
53 .6 ±
21 .2
99 .7 ±
16 .4
23 .7
1. 6 ±
0. 6
4. 8 ±
2. 9
5. 8 ±
2. 3
Walker2d
B E
A R
B R
A C
-p A
W R
R an
do m
7. 3
1. 5 ±
0. 9
3. 1 ±
0. 9
3. 2 ±
0. 4
-0 .2
0 ±
0. 2
1. 3 ±
0. 7
3. 2 ±
2. 5
1. 5
1. 3 ±
0. 4
2 ±
1 2.
4 ±
0. 8
M ed
iu m
59 .1
-0 .5 ±
0. 3
0. 6 ±
0. 5
0. 3 ±
0. 7
77 .5
6. 4 ±
9. 9
70 ±
10 .1
78 ±
3. 1
17 .4
14 .8 ±
2. 8
17 .1 ±
0. 2
17 .2 ±
0. 1
M ed
iu m
-R 19
.2 0.
7 ±
0. 6
6. 5 ±
5. 1
7. 3 ±
1. 3
-0 .3
8. 5 ±
2. 2
9. 9 ±
2 18
.6 ±
6. 5
15 .5
7. 4 ±
2. 1
1. 6 ±
0. 4
1. 5 ±
0. 3
M ed
iu m
-E 40
.1 -0
.1 ±
0. 1
1. 5 ±
2. 5
2. 3 ±
2. 2
76 .9
20 .6 ±
16 .8
64 .1 ±
10 .8
77 .5 ±
3. 1
53 .8
35 .5 ±
10 .4
52 .5 ±
1. 2
53 .3 ±
0. 3
Walker2d
B C
Q C
Q L
M O
PO
R an
do m
4. 9
1. 8 ±
0. 9
4. 5 ±
0. 5
4. 8 ±
0. 3
7 1.
7 ±
1. 3
3. 2 ±
1. 4
3. 4 ±
1. 9
13 .6
-0 .2 ±
0. 2
-0 .1 ±
0. 1
-0 .1 ±
0. 2
M ed
iu m
53 .1
32 .8 ±
8. 2
50 .9 ±
4. 3
52 .3 ±
1. 4
79 .2
42 .9 ±
24 .2
80 ±
1. 2
81 .7 ±
3. 1
17 .8
7 ±
3. 6
5. 7 ±
4. 7
11 ±
4. 3
M ed
iu m
-R 15
6. 9 ±
0. 6
14 .9 ±
0. 2
15 .1 ±
0. 2
26 .7
4. 6 ±
3. 9
0. 8 ±
0. 5
2 ±
1. 5
39 5.
1 ±
5. 7
3. 1 ±
2. 4
14 .2 ±
4. 5
M ed
iu m
-E 57
.5 32
.5 ±
9. 1
55 .2 ±
3. 8
57 .2 ±
0. 2
11 1
49 .5 ±
26 .7
63 .5 ±
22 .5
93 .3 ±
8. 8
44 .6
5. 3 ±
3. 9
5. 5 ±
3. 5
17 .2 ±
8. 7
Under review as a conference paper at ICLR 2022
Ta bl
e 22
:N or
m al
iz ed
sc or
es fo
rt he
D 4R
L ta
sk s
(w ith
jo in
tn oi
se sh
if t)
.W e
ta ke
th e
ba se
lin e
re su
lts (f
or 10
T )o
fM O
PO fr
om th
ei ro
ri gi
na lp
ap er
s an
d th
at of
th e
ot he
rm od
el -f
re e
m et
ho ds
(B E
A R
,B R
A C
-p ,A
W R
,B C
Q an
d C
Q L
)f ro
m th
e D
4R L
pa pe
r( Fu
et al
., 20
20 ).
Jo in
tN oi
se Sh
if t
10 T
1T 1T
+1 0S
1T +1
0S 10
T 1T
1T +1
0S 1T
+1 0S
10 T
1T 1T
+1 0S
1T +1 0S w /o A ug . (D A R A ) w /o A ug . (D A R A ) w /o A ug . (D A R A
)
Hopper
B E
A R
B R
A C
-p A
W R
R an
do m
11 .4
0. 6 ±
0 7.
4 ±
0. 5
4. 2 ±
3. 6
11 10
.8 ±
0. 2
10 ±
0. 8
10 .8 ±
0 10
.2 10
.1 ±
0 3.
6 ±
0 4 ±
0. 4
M ed
iu m
52 .1
0. 8 ±
0 2 ±
1 2 ±
0. 1
32 .7
26 .6 ±
4. 8
27 .6 ±
2. 8
37 .6 ±
7 35
.9 30
.3 ±
0. 1
38 .8 ±
3. 9
41 .3 ±
5. 4
M ed
iu m
-R 33
.7 2.
7 ±
1. 6
3. 6 ±
0. 4
9. 9 ±
6. 3
0. 6
13 .4 ±
6. 4
89 .9 ±
7. 8
10 1.
4 ±
0. 2
28 .4
12 .4 ±
6 6.
7 ±
4. 2
7. 2 ±
0. 2
M ed
iu m
-E 96
.3 0.
8 ±
0. 2
0. 8 ±
0 1.
4 ±
0. 6
1. 9
19 .8 ±
14 57
.6 ±
23 .4
87 .8 ±
13 .3
27 .1
25 .5 ±
1. 2
27 ±
0 27 ±
0. 1
Hopper
B C
Q C
Q L
M O
PO
R an
do m
10 .6
10 .5 ±
0. 1
7 ±
0 9.
6 ±
0 10
.8 10
.4 ±
0. 1
10 .4 ±
0. 4
10 .8 ±
0 11
.7 1.
5 ±
0. 8
1. 3 ±
0. 5
2. 9 ±
1. 5
M ed
iu m
54 .5
45 .8 ±
2. 2
49 ±
1. 7
54 .4 ±
0. 1
58 46
.2 ±
11 .9
58 ±
0 58 ±
0 28
2. 7 ±
2. 1
9. 2 ±
5. 4
17 .3 ±
3. 4
M ed
iu m
-R 33
.1 13 ±
5 23
.8 ±
3. 2
32 ±
0. 9
48 .6
13 .6 ±
6. 4
2. 6 ±
0. 3
3. 6 ±
0. 6
67 .5
0. 8 ±
0. 1
2. 3 ±
1. 7
6. 4 ±
0. 8
M ed
iu m
-E 11
0. 9
44 .6 ±
18 .6
96 ±
0. 5
10 9 ±
0. 2
98 .7
50 .7 ±
26 .9
73 .4 ±
1. 5
10 8.
9 ±
0. 7
23 .7
1 ±
0. 2
6. 1 ±
1. 4
7. 5 ±
0. 6
Walker2d
B E
A R
B R
A C
-p A
W R
R an
do m
7. 3
2. 2 ±
0. 1
0. 6 ±
0. 1
2. 6 ±
0. 3
-0 .2
2. 8 ±
2. 8
3. 3 ±
2. 9
8. 8 ±
8 1.
5 0.
9 ±
0. 1
1. 5 ±
0. 6
1. 5 ±
0. 2
M ed
iu m
59 .1
-0 .4 ±
0. 1
0. 6 ±
0 0.
1 ±
0. 3
77 .5
28 .8 ±
28 .4
55 .2 ±
15 .8
72 .9 ±
9. 1
17 .4
12 .2 ±
0. 3
17 .2 ±
0. 2
17 .2 ±
0. 2
M ed
iu m
-R 19
.2 0.
4 ±
0. 2
4 ±
0. 2
10 .4 ±
2. 4
-0 .3
6. 3 ±
1. 2
32 .1 ±
11 .9
34 .8 ±
10 .5
15 .5
6 ±
1 1.
4 ±
0 2.
1 ±
0. 9
M ed
iu m
-E 40
.1 -0
.2 ±
0. 2
0. 8 ±
0. 4
0. 6 ±
0. 6
76 .9
21 .8 ±
18 .4
62 .3 ±
13 .1
74 .3 ±
1. 8
53 .8
40 .4 ±
12 .6
53 ±
0. 1
53 .6 ±
0
Walker2d
B C
Q C
Q L
M O
PO
R an
do m
4. 9
3. 7 ±
1. 8
3. 4 ±
0. 4
5. 2 ±
0. 3
7 0.
5 ±
1 2.
7 ±
0. 2
6. 4 ±
0. 6
13 .6
-0 .3 ±
0. 1
-0 .2 ±
0 -0
.2 ±
0. 2
M ed
iu m
53 .1
43 ±
8. 3
44 .9 ±
3. 3
52 .7 ±
0. 3
79 .2
43 .9 ±
21 .7
73 .2 ±
0. 8
81 .2 ±
1. 1
17 .8
5. 8 ±
5. 9
7. 8 ±
6. 2
12 .2 ± 5 M ed iu m -R 15 5. 7 ± 0. 5 9. 8 ± 5. 2 14 .6 ± 0. 4 26 .7 1. 8 ± 1. 2 1. 4 ± 0. 4 1. 8 ± 0. 4 39 0. 8 ± 0. 7 9. 3 ± 5. 2 16 .4 ±
4. 9
M ed
iu m
-E 57
.5 44
.5 ±
3. 6
40 .6 ±
16 .4
57 .2 ±
0. 2
11 1
46 .8 ±
40 10
9. 9 ±
4. 5
11 6.
5 ±
9. 1
44 .6
2. 9 ±
3. 1
15 .2 ±
12 .8
26 .3 ±
18 .4

Review summaries:
The given work proposes the addition of a dynamics aware framework for Offline Reinforcement Learning. It motivates that  offline reinforcement learning suffer from a dynamics shift in SARSA transitions, which need to be effectively handled. They propose modifying reward in source offline dataset, penalizing it with the dynamics aware reward transition, setting the lower bound. The reward augmentation is done with explicit policy constraints as well as implicit policy constraints.They evaluate this approach on various robotic manipulation task from the Gym-MujoCo suite, evaluating and showcasing perforomance on four offline RL algorithms.  

This paper presents a new offline RL algorithm that aims to adapt the policy to a target domain with insufficient data with the help of abundant data in a source domain. The authors propose their method, DARA, which applies a reward penalty that correct the dynamics shift between the source and target domains. The authors combine DARA with previous model-based and model-free offline RL methods and show that DARA is able to achieve performance improvement over prior offline RL methods without the reward augmentation in both simulated and real-world tasks.

Offline reinforcement learning could enable one to train a data-driven decision making engine from purely offline data. However, the performance of polices learned through existing offline RL methods heavily depend on the quantity and quality of the offline data. When the data is scarce for a task, existing offline RL methods trained on limited offline data could perform poorly.   This work tackles the data efficiency issue of offline RL by transferring knowledge from a source offline dataset to a target offline dataset following an environment dynamics different from the source offline dataset. The proposed method augments the reward function by penalizing the state-action-next state transition tuples that are rare in the source environment. This reward augmentation strategy directly accounts for the dynamics shift between the source and the target domain. On several simulation a real-world tasks, the proposed method significantly improved the data-efficiency of existing offline RL algorithms. 

The paper focuses on the dynamics shift problem in the offline reinforcement learning setting, with the assumption that the estimated MDP from the offline dataset is deterministic. It proposes DARA, a framework that relaxes the requirement of data (both the amount of data and the reward information contained in the data) needed from the target task. By capturing the dynamics shift between the source and target environments, DARA modifies the reward gained in the source environment to discourage transitions that have a smaller probability to happen in the target environment. The framework works with both model-free and model-based settings.