DARA: Dynamics-Aware Reward Augmentation in Offline Reinforcement Learning
Authors: Jinxin Liu
Zhang Hongyin
Donglin Wang
Venue: ICLR 2022 Poster (2022)

Abstract: Offline reinforcement learning algorithms promise to be applicable in settings where a fixed dataset is available and no new experience can be acquired. However, such formulation is inevitably offline-data-hungry and, in practice, collecting a large offline dataset for one specific task over one specific environment is also costly and laborious. In this paper, we thus 1) formulate the offline dynamics adaptation by using (source) offline data collected from another dynamics to relax the requirement for the extensive (target) offline data, 2) characterize the dynamics shift problem in which prior offline methods do not scale well, and 3) derive a simple dynamics-aware reward augmentation (DARA) framework from both model-free and model-based offline settings. Specifically, DARA emphasizes learning from those source transition pairs that are adaptive for the target environment and mitigates the offline dynamics shift by characterizing state-action-next-state pairs instead of the typical state-action distribution sketched by prior offline RL methods. The experimental evaluation demonstrates that DARA, by augmenting rewards in the source offline dataset, can acquire an adaptive policy for the target environment and yet significantly reduce the requirement of target offline data. With only modest amounts of target offline data, our performance consistently outperforms the prior offline RL methods in both simulated and real-world tasks. 

Published as a conference paper at ICLR 2022
DARA: DYNAMICS- AWARE REWARD AUGMENTATION IN OFFLINE REINFORCEMENT LEARNING
Jinxin Liu 123+ Hongyin Zhang!* Donglin Wang!
1 Westlake University. Zhejiang University.
3 Institute of Advanced Technology, Westlake Institute for Advanced Study. (liujinxin, zhanghongyin, wangdonglin) @ westlake.edu.cn
ABSTRACT
Offline reinforcement learning algorithms promise to be applicable in settings where a fixed dataset available and no new experience can be acquired. How- ever, such formulation is inevitably offline-dat--uungyy and, in practice, collecting a large offline dataset for one specific task over one specific environment also costly and laborious. In this paper, we thus 1) formulate the offline dynamics adaptation by using (source) offline data collected from another dynamics to relax the requirement for the extensive (target) offline data, 2) characterize the dynamics shift problem in which prior offline methods do not scale well, and 3) derive a sim- ple dynamics-aware reward augmentation (DARA) framework from both model- free and model-based based offline settings. Specifically, DARA emphasizes learning from those source transition pairs that are adaptive for the target environment and mitigates the offline dynamics shift by characterizing paat--actionnextt nextisttteett-tteee pairs instead the typical state-action distribution sketched by prior offline RL meth- ods. The experimental evaluation demonstrates that DARA, by augmenting re- wards in the source offline dataset, can acquire an adaptive policy for the target environment and yet significantly reduce the requirement of target offline data. With only modest amounts of target offline data, our performance consistently outperforms the prior offline RL methods in both simulated and real-world tasks.
INTRODUCTION
Offline reinforcement learning (RL) (Levine et al. 2020 Lange
Normalized score
al. 2012), the task of learning from the previously collected dataset, holds the promise of acquiring policies without any costly active interaction required in the standard online RL paradigm. However, we note that although the active trail-and-error (online exploration) is eliminated, the performance of offline RL method heavily relies on the amount of offline data that is used for training.
100
BEAR
CQL
50-
MOPO
100%
50%
20% 10%
5%
Amount of data used training
As shown in Figure]] I the performance deteriorates dramatically as the amount of offline data decreases. A natural question therefore arises: can we reduce the amount of the (target) offline data without significantly affecting the final performance for the target task?
Figure 1: Solid and dashed lines denote offline Medium- Replay and Medium-Expp Expert data in D4RL (Walker2d) resp.
Bringing the idea from the transfer learning (Pan & Yang/ 2010), we assume that we have access to another (source) offline dataset, hoping that we can leverage this dataset to compensate for the performance degradation caused by the reduced (target) offline dataset. In the offline setting, pre- vious work (Siegel al.] 2020 Chebotar al., 2021) has characterized the reward (goal) differ- ence between the source and target, relying on the "conflicting" or multi-goal offline dataset (Fu) al. 2020),, while we focus on the relatively unexplored transition dynamics difference between the source dataset and the target environment. Meanwhile, we believe that this dynamics shift is not arbitrary in reality: in healthcare treatment, offline data for particular patient is often limited, whereas we can obtain diagnostic data from other patients with the same case (same reward/goal)
"quual contribution.
Corresponding author.

---

Published as a conference paper at ICLR 2022
and there often exist individual differences between patients (source dataset with different transition dynamics). Careful treatment with respect to the individual differences is thus a crucial requirement.
Given source offline data, the main challenge is to cope with the transition dynamics difference, i.e., strictly tracking the state-action supported by the source offline data can not guarantce that the same transition state-action-nexxt -tettttn-eett-tttee can be achieved in the target environment. However, in the offline setting, such dynamics shift is not explicitly characterized by the previous offline RL methods, where they typically attribute the difficulty of learning from offline data to the state-action distribution shift (Chen & Jiang 2019; Liu et al. 2018). The corresponding algorithms (Fujimoto et al., 2019) Abdolmaleki et al. 2018 Yu al. 2020) that model the support of state-action distribution induced by the learned policy, will inviiaably suffer from the transfer problem where dynamics shift happens.
Our approach is motivated by the well established connection between reward modification and dynamics adaptation (Kumar et al.) 2020b] Eysenbach & Levine 2019; Eysenbach et al.) 2 which indicates that, by modifying rewards, one can train policy in one environment and make the learned policy to be suitable for another environment (with different dynamics). Thus, we propose besides characterizing the state-action
to exploit the joint distribution of state-action-nnex-stee distribution shift as in prior offlin RL algorithms, we additionally identify the dynamics (i.e., the conditional distribution of next-state given current state-action pair) shift and penalize the agent with dynamics-aware reward modification. Intuitively, this reward modification aims to discourage the learning from these offline transitions that are likely in source but are unlikely in the target environment. Unlike the concurrent work (Ball al. 2021 Mitchell et al. (021) paying attention to the offline domain generalization, we explicitly focus on the offline domain (dynamics) adaptation.
Our principal contribution in this work is the characterization of the dynamics shift in offline RL and the derivation of dynamics-aware reward augmentation (DARA) framework built on prior model- free and model-based formulations. DARA is simple and general, can accommodate various offline RL methods, and can be implemented in just few lines of code on top of dataloader at training. In our offline dynamics adaptation setting. we also release a dataset, including the Gym-MuJoCo tasks (Walker2d, Hopper and HalfCheetah), with dynamics (mass, joint) shift compared to D4RL,, and a 12-DoF quadruped robot both simulator and real-world. With only modest amounts of target offline data, we show that DARA-based offline methods can acquire adaptive policy for the target tasks and achieve better performance compared to baselines both simulated and real-world tasks.
2 RELATED WORK
Offline RL describes the setting in which learner has access to only fixed dataset of experience, while no interactive data collection is allowed during policy learning (Levine et al., 2020). Prior work commonly assumes that the offline experience is collected by some behavior policies on the same environment that the learned policy be deployed on. Thus, the main difficulty of such offline setting is the state-action distribution shift (Fujimoto et al.] 2019 Liu et al. 2018). Algorithms address this issue by following the two main directions: the model-free and model-based offline RL.
Model-free methods for such setting typically fall under three categories: 1) Typical methods mit- igate this problem by explicitly (Fujimoto et al. 2019 Kumar et al., 2019 Wu et al. (019)) or implicitly (Siegel et al., 22200: Peng al. 2019 Abdolmaleki et al.) 2018) constraining the learned
policy away from OOD state-action pairs. 2) Conservative estimation based methods learn pes- simistic value functions to prevent the overestimation (Kumar al. 2020a Xu et al. 2021). 3)
Importance sampling based methods directly estimate the State-marginn importance ratio and ob- tain an unbiased value estimation (Zhang et al., 2020 Nachum & Dai 2020 Nachum et al. 2019b).
Model-based methods typically eliminate the state-action distribution shift by incorporating reward penalty, which relies on the uncertainty quantification of the learned dynamics (Kidambi et al., 2020) Yu et al.] 2020). To remove this uncertainty estimation, Yu et al. (2021) learns conservative critic function by penalizing the values of the generated state-action pairs that are not in the offline dataset.
These methods, however, define their objective based on the state-action distribution shift, and ig- nore the potential dynamics shift between the fixed offline data and the target MDP. In contrast, we account for dynamics taatte--ctionneex---ee ward augmentation. A counterpart, close to our work, is off-dynamics RL (Eysenbach et al. 202I), where they set up dynamics shift in the interactive environment while we focus on the offline setting.
shift and explicitly propose the dynamics aware
2

---

Published as a conference paper at ICLR 2022
3 PRELIMINARIES
We study RL in the framework of Markov decision processes (MDPs) specified by the tuple M (S, A. r; T, Po: 7), where S and A denote the state and action spaces, r(s a) E [- -Raar, Rmaz] is the reward function, T(s'|s, a) is the transition dynamics, po(s) is the initial state distribution, and is the discount factor. The goal in RL is to optimize a policy π (als) that maximizes the expected discounted return 77M (T) := ET (+) 1i 7 7'r(St, at)], where T = (So, ao, S1, a1. ...). We also define Q-values Q(s, a) == Er-pI,(r) (T) 15==0 7'r(s,, |5) = S, ag = al, V-values V(s) = (a-r(al=)) (2,, a)], and the (unnormalized) state visitation distribution di (s) = Ei= 7 P(S|T, M, where P(S|TT, () denotes the probability of reaching state s at time / by running и in M.
In the offline RI. problem, we are provided with a static dataset 'D
a, s')}, which con- sists of transition tuples from trajectories collected by running one or more behavioral policies, denoted by Th. on MDP M. With slight abuse of notation, we write D {(s, a, a, s') dp (T) (a (s)s)sss a) [s'|ss, a)}, where the (s) denotes state-marginal distribution in D. In the offline setting, the goal is typically to learn the best possible policy using the fixed offline dataset.
Model-free RL algorithms based on dynamic programming typically perform policy iteration to find the optimal policy. Such methods iteratively conduct 1) policy improvement with Ga := arg max_ x Ex (())s)) [Q(s, a)] and 2) policy evaluation by iterating the Bellman equation ((s, B% ( a) := r(s,a) + ("""s,)))))))s)..)) (<>">("'ss") [Q(s', a')] over dīr (S))(a(s)) Given off-ppolict D, we resort to 1) improvement with GDQ == arg max, (S) [Q (s a)] and 2) evaluation by iterating Q(s, a) = B% (s, a) == r(s, a) +
[Q a²)]
over all (s, a) in D. Specifically, given any initial Qo, it iterate
Policy improvement: - k++ GDQ",
Policy evaluation: Q²+1= = BD Q²
(1)
Model-free offline RL based on the above iteration suffers from the state-action distribution shift, Le., policy evaluation B% Qk-1 may encounter unfamiliar state action regime that is not covered by the fixed offline dataset D, causing erroneous estimation of Q². Policy improvement GoQk further exaggerates such error, biasing policy -=1 towards out-of-distributio (OOD) actions with erroneously high Q-values. To address this distribution shift, prior works 1) explicitly constrain
policy be close to the behavior policy (Fujimoto al. Ghasemipour et al. 2021). introducing penalty o D π
2019: Kurnar et al. 2019 Wu et al. 2019
8), To (a)s into Go or D in Equation
GDQ arg max E. (e-do(s) Q(ss. a) "
a ma
(5Q a) = r(s, a) + B (s
[Q(s', aD Th (aa)sss)))
where D is a divergence function between distributions over actions (e.g.. MMD or KL divergence). or 2) train pessimistic value functions (Kumar et al. 2020a Yu et al. 2021 Xu al. 2021), penal- izing Q-values at states in the offline dataset D for actions generated by the current policy n:
Q = arg min (ss [Q(s, a)], s.t. Q = B5Q.
(3)
Q
Model-based RL algorithms iteratively 1) model the transition dynamics T(s' [s, a), using the data collected in M: max; E.g.a.s (s) (s) [log ( |s, a)]. and 2) infer policy æ from the modeled M (S,A, TT, Po. 7). where assume that and po are known, maximizing 7 (T) with a planner or the Dyna-style algorithms (Sutton 1990). In this paper, we focus on the latter.
Mosel-based offline RL algorithms similarly suffer from OOD state-action (Kidambi et al. 2020
(log Like the conservative estimation approach described in Equation 3 recent conservative model-basee
Cang al. 2021) if we directly apply policy iteration over ी == max.i Ex,,,''> [log T(s' (s, a)]
offline RL methods provide the policy with a penalty for visiting states under the estimated 7 where I is likely to be incorrect. Taking u(s, a) as the oracle uncertainty (Yu et al. (220)) that provides a consistent estimate of the accuracy of model I at (s, a), we can modify the reward function to obtain conservative MDP: M. = AS, au, T, Po; 7), then learn a policy π by maximizing 7grr. (T).
For parametric Q-fucttion, we often perform @+ ++ f- arg ming E( ++1 Q* -Qsss
3

---

Published as a conference paper at ICLR 2022
4 PROBLEM FORMULATION
In standard offline RL problem, the static offline dataset D consists of samples {(s, a, r,s') dp (m) (a)s)r(s, (s'ss a)}. Although offline RL methods learn policy for the target MDP M == (S,A, r, T, :Po: 7) without (costly) online data, we shown in Figure [ it requires fair amount of (target) offline data D collected on M. Suppose we have another (source) offline dataset D', consisting of samples {(s, a,,"' ~ d (s)
(s"' (s. a)} collected by the behav- ior policy TV on MDP M' == AS, r, Po, 7), then we hope the transfer of knowledge between offline dataset (D' UD) can reduce the data requirements on D for learning policy for the target M.
4.1 DYNAMICS SHIFT IN OFFLINE RL
Although offline RL methods in Section 3 have incorporated the state-action distribution constrained backups (policy constraints or conservative estimation), they also fail to learn an adaptive policy for the target MDP M with the mixed datasets (D' U UD). J. as we show in Figure | (Appendix). We attribute this failure to the dynamics shift (Definition between D' and M in this adaptation setting.
(Empirical MDP) An empirical MDP estimated from D is M == Ar, r, T, po. 7) where T- max; s' [log (s'(s, a)] and (s'sss for all (s, a, s') not in dataset D.
Definition 1
po:7) be the empirical MDP estimated from D. To evaluate a policy π for M == (S,A,r, T. Pu: 7) with offline dataset D, we say that the dynamics shift (between D and M) in offtine RL happens if there exists at least transition pair as)a.s') s')) dh (s) (als)T a) 0} such that ( a) # T(s (s. a).
Definition 2 (Dynamics shift) Let M ==
In practice, for a stochastic M and any finite offline data D collected in M, there always exists the dynamics shift. The main concern is that finite samples are always not sufficient to exactly model stochastic dynamics. Following Fujimoto et al. 2019), we thus assume both MDPs M and M' are deterministic, which means the empirical M and M' are both also deterministic. More importantly, such assumption enables us to explicitly characterize the dynamics shift under finite offline samples.
Lemma Under deterministic transition dynamics, there is no dynamics shift between DD and M.
For offline RL tasks, prior methods generally apply BBQ along with the state-action distribution correction (Fquations]) 2 and 3.. which overlooks the potential dynamics shift between the (source) offline dataset and the target MDP (e.g.. D' + M.). As a result, these methods do not scale well to the setting in which dynamics shift happens, e.g., learning an adaptive policy for M with (source) D'.
4.2 DYNAMICS SHIFT IN MODEL-FREE AND -ASEL OFFLINE FORMULATIONS
From the model-free (policy iteration) view, an exact policy evaluation on M is characterized by iterating Q(s, a) 3% (s,, a) for all (s, a) such that d (T) (als) > 0. Thus, to formalize the pol- icy evaluation with offline D or D' (for an adaptive æ on target M), we require that Bellman operator (B, a) or B3, ( a) approximates the oracle 8i (s a) for all (s, a) in Sx or S", where S, and S denote the sets ((s, a): dp (s) > 0} and {(s, a) : d (s) J (a s) > 0} respectively.
1) To evaluate a policy æ for M with D (i.e., calling the Bellman operator B3), notable model- free offline method BCQ (Fujimoto et al. 2019) translates the requirement of B% = =B,, into the requirement of (s'(s, a) = T(s'|s, a). Note that under deterministic environments, we have the property that for all (s. s') in offline data D. (s''s,, a) [s'' a) (Lemmal) 1.. As a result, such property permits BCQ to evaluate a policy æ by calling B5, replacing the oracle B% meanwhile constraining S~ to be subset of the support of (s) (als). This means policy и which only traverses transitions contained in (target) offline data D, can be evaluated on M without error.
2) To evaluate a policy for M with D' (i.e., calling the Bellman operator B%,) ) we have lemma
Lemma 2 Dynamics shift produces that B (s, a) + Bi, ((s, a) for some (s, a) in St.
With the offline data D', lemma 2 suggests that the above requirement B7, = B becomes infea- sible, which limits the practical applicability of prior offline RL methods under the dynamics shift.
4

---

Published as a conference paper at ICLR 2022
To be specific, characterizing adaptive policy for target MDP M with D' moves beyond the reach of the off off-policy evaluation based on iterating Q = 1% Q (Equations 2
and 3). Such iteration may
cause the evaluated Q (or learned policy л) overfits to T' and struggle to adapt to the target T. To overcome the dynamies shift, we would like to resort an additional compensation Aprr such that
B (s, a) + a) B% Q a)
(4)
for all (s, a) in S. Thus, we can apply B + Aprr to act as a substitute for the oracle Q.
From the model-based view, the oracle 7ar(() (calling the Bellman operator Bi on the target M) and the viable x (T) (calling 8 on the estimated M' from source D') have the following lemma.
Lemma 3 Let B (s) = (a)=(()))) (r(s,a) a) - - (..) [V (V('s)) [s'] For any n,, we have:
Ma (T) = MM (I) + Egndy (x) [B% VMr(s) Bi (s)].
Mr Vsr(s) - B Var (s)] I c, Tas (T) will be improved. If F is a set of functions f:S + R that contains Vas. then we have
Lemma states that if we maximize (T)) subject (s)
1
E Eg (s)
BT VM((s) - B% VM VM (s)]
S 3B (m)m()))))
dd a), a)s'ss (5)
where d (T"("sss' a), (ss a)) supjer ( -Fgs))]-FssTTollaaa [f(s') I. which is the integral probability metric (IPM). Note that if we directly follow the admissible error assumption in MOPO (Yu et al. 2020) i.e., assuming dF'("' (s'ls, [s, a), T(''ss ()sss))) a)) I u(s, a) for all (s, a), this would be too restrictive: given that 7" is estimated from the source offline samples collected under T", not the target T, thus such error would not decrease as the source data increases. Further, we find
(6) Thus, we can bound the dx T') term with the admissible error assumption over d> T), as in MOPO, and the auxiliary constraints dT"
(s' |s, a), (s' a)) < d> (s'ss, a), (s'ls, a)) + d (s'ls (s.ss,s.ee a), ((s'ss,))
( 7). See next section for the detailed implementation.
In summary, we show that both prior offline model-free and model-based formulations suffer from the dynamics shift, which also suggests us to learn a modification (Á or dy:) to eliminate this shift.
DYNAMICS-AWARE REWARD AUGMENTATION
In this section, propose the dynamics-aware reward augmentation (DARA), simple data aug- mentation procedure based on prior (model-free and model-based) offline RL methods. We first provide an overview of our offline reward augmentation motivated by the compensation Airr in
Equation A and the auxiliary constraints dT", 7) in Equation 6 and then describe its theoretical derivation in both model-free and model-based formulations. With the (reduced) target offline data D and the source offline data D', summarize the overall DARA framework in Algorithm
Algorithm 1 Framework for Dynamics Dynamics-Aware Reward Augmentation (DARA)
Require: Target offline data D (reduced) and source offline data D'
I: Learn classifiers (gas and qu) that distinguish source data D' from target data D. (See Appendix A.1.3
2: Set dynamics synaa aware Ar(se, as, St+1) log Teas (target (c4-1)
(sourcejs; (1.8))1
log (somrce os.)
3: Modify rewards for all (8c, ae; re: Be+1) D': re +- rs ndr.
4: Learn policy with (D U D') using prior model-free or madel-based offline RI algorithms.
5.1 DYNAMICS-AWARE REWARD AUGMENTATION MODEL-FRET FORMULATION
Motivated by the well established connection of RL and probabilistic inference (Levine) 2018 we first cast the model-free RL problem as that of inference in a particular probabilistic model. Specifically, introduce the binary random variable O that denotes whether the trajectory
5

---

Published as a conference paper at ICLR 2022
(su, ag, $1; is optimal (O 1) not (O = 0). The likelihood of a trajectory then be modeled as p0 = 1/7) = exp (E, rt/n), where T; := r(s,, a.) and 1j > is a temperature parameter.
(Reward Augmentation with Explicit Policy/Value Constraints) We now introduce variational distribution Pyg. (7) ) p(so) I (s $,, ar) ( (S)) to approximate the posterior distribution
Ple (T|O- 1), which leads the evidence lower bound of log PM (O 1):
PM (T)
log log (O = 1) = log E, (+) [p(O- = 1|T)] 2 E Br-ppy
log logp(O =1/7) + log D (T)
ri/n re/n- log T" (S($ (8:, (s,,ar) ar)
Σ
= Byppp Sir (T)
T(s++1 St, at)
Since we are interested in infinite horizon problems, we introduce the discount factor y and take the limit of steps in each rollout, i.e., H + 00. Thus, the RL problem on the MDP M, cast as the inference problem arg max, log (Or((0 1), can be stated as a maximum of the lower bound Erap (7) Ei=? 7 (re log (B: T (a.
This is equivalent to an RL problem on A/' with (s'sss.))
the augmented reward r <- r(s, a) - 7 log (.s"s Intuitively, the lo ('ssss..))
((s'ssa) term discour- in D' that have low transition probability in the target M.
ages transitions
In the model-free offline setting, we can add the explicit policy or Q-value constraints (Equations- 2 and 3) to mitigate the OOD state-actions, Thus, such formulation allows the oracle B to be re- expressed by B7, and the modification log +. which makes the motivation in Equation practical.
(Reward Augmentation with Implicit Policy Constraints) If we introduce the variational distri- (T) (T)
bution p
p(so) I (' (S |s,, aT) (a, (sa), can recover the weighted-
style (Wang al. 2020 Peng al. 2019 Abdolmaleki et al. 2018 Peters et al. 2010) objective by
maximizing [['', T) = E
Tropp (7)
Di= 7' Tr log
7(#+41 (R....) 17 log
[8111 (8.1))
x [((alar)) (lower <(ae[s)))
bound of log (O = 1)). Following the Expectation Maximization (EM) algorithm, we can maxi- mize J [ α) by iteratively (E-step) improving [(T',)) w.r.t. n' and (M-step) updating π w.r.t. n².
(r) Enn" e log T(*(Ia.)) (so =s, a, = a, s' Then, given offline data D', rewrite I (TT,.) as constrained objective (Abdolmaleki et al.
(E-step) We define Q(s, a,s') = Eyappy
('('
2018):
max E de (al) [Q(ss,) - (s,,
s.t. (xgd [KL ( (alss] Se.
When considering a fixed π. the above optimization over n' can be solved analytically (Vieillard al., 2020; Geist et al.) 2019 Peng al. 2019). The optimal n.. is then given by ( oc T (als) exp (Q(s, a)) exp( (s. a,
T' (s's'ss, |s, a))). As the policy evaluation in Equation (Footnote-
2), we estimate Qs,, a) and Q(s, a, s') by minimizing the Bellman error with offline samples in D'.
(M-step) Then, we can project TI, onto the manifold of the parameterized w:
arg arg min
Egndp (x) [DLL (als)
arg max [log BR log
(als) exp (Q(s, a)) exp (-n@(s,a,s)))
(8)
From the regression view, prior work MPO (Abdolmaleki al. 2018) infers actions with Q-value weighted regression, progressive approach compared to behavior cloning; however, such paradigm lacks the ability to capture transition dynamics. We explicitly introduce the exp(-n)(ss, a, s')) term, which as we show in experiments, is crucial component for eliminating the dynamics shift.
Implementation: In practice, we adopt offline samples in 'D to approximate the true dynamics T of M, and introduce pair of binary classifiers, ( (-|s, a, a,s') and (-1s,, a). to replace log
7"' (s' (s's(18.a)
[(s' as
in Eysenbach et al. (2021): ; log T(*|s.a)
('()'ss.a) log gua (source/s.a.s')
qus (target(s.n.55) (s.,,s) log (=(soure|ss.a) (See Appendix- a. (aftgrget)s.n)
A. 1.3 for detauls). Although the amount of data D sampled from the target M is reduced in our prob-
lem setup, experimentally find that such classifiers are sufficient to achieve good performance.

---

Published as a conference paper at ICLR 2022
5.2 DYNAMICS-AWARK REWARD AUGMENTATION MODEL-BASED FORMULATION
Following Equation 6 we then characterize the dynamics shift compensation term as in the above model-free analysis in the model-based offline formulation. We will find that across different deriva- tions, our reward augmentation Ar has always maintained the functional consistency and simplicity.
Following MOPO, assume F {f I I 1), then we have dz df a), [s'ss, a)) (Tv( |s. a). (s' is, a)) S (Daa s. a), (s'|s, 22)/ where Drv is the total variance distance. Then we introduce the admissible error u(s, a) such that ()=((s'|ss fs| a), ((s'|s,))) S
u(s, a) for all (s, a), and 1/ and 5 such that (DRI(T", T)22) I 7DKr. (T", T) + 6. Following Lemma 3 we thus can maximize the following lower bound with the samples in M' (A == RBmux) )
('ssss. Au(s, a) Ao as)
(T) 2 E (o)r(aa))
r(s.a) 11
(9)
Implementation: We model the dynamics f" and T with an ensemble of N parameterized Gaus- sian distributions: No.. (Ho a) a)) and NV.. (Ho Eg, a)). where i E [L.N]. We approximate u with the maximum standard deviation of the learned models in the ensemble: usss a) = max 1 | (s. a)Ilr, omit the iinnning-iddeeedeentt 6, and treat À as a hyperparameter as in MOPO, For the log 2' term, we resort to the above classifiers (ysas and qsa) in model-free set- ting. (See Appendix A.3.2 for comparison between using classifiers and estimated- estimated-dynamice ddnnamiis ratio.)
6 EXPERIMENTS
We present empirical demonstrations of our dynamics-aware reward augmentation (DARA) in a variety of settings. We start with two simple control experiments that illustrate the significance of DARA under the domain (dynamics) adaptation setting. Then we incorporate DARA into state-of- the-art (model-free and model-based) offline RL methods and evaluate the performance on the D4RL tasks. Finally, we compare our framework to several cross-domain- and real-world tasks. Note that for the dynamics adaptation, we also release (source) dataset as a complement to D4RL, along with the quadruped robot dataset in simulator (source) and real (target).
baselines on simulated
6.1
How DOES DARA HANDLE THE DYNAMICS SHIFT IN OFFLINE SETTING?
tource/target
source
target
Q (wit Aug.) (DARA)
gooal
State[11]]
-0.26 0.26
DAR A
(53, 211) (5ig ayl
Rastrictoti in target
Time step
Figure 3: Internal dynamics shift: (left) source and target MDPs (range of the ragkt-back-leg of the ant (state[11]) is limited: [-0.52, 0.52] in source MDP + [-0.26, 0.26] in target MDP); (right) the solid (orange) line denotes the state of the right-back-let over one trajectory col- lected in source, dashed (blue) line denotes the learned reward modification - -Ar over the tra- jectory, and green and red slices denote transi- tion pairs where - -Ar 2 and - -Ar < 0, resp.
Figure 2: External dynamics shift: (left) source and target MDPs (target contains obstacle rep- resented with the dashed line); (middle) top plots (w/o Aug.) depict the trajectories that gen- crated by the learned policy with vanilla MPO; (middle) bottom plots (DARA) depict the tra- jectories that are generated by the learned pol- icy with DARA-based MPO; (right) learned Q- values on the state-action pairs in left subfigure.
Here characterize both external and internal dynamics shifts: In Map tasks (Figure leett, the source dataset D' is collected in a 2D map and the target D is collected in the same environment but with obstacle (the dashed line); In Ant tasks F the source dataset D' is collected using the Mujoco Ant and the target D is collected with the same Ant but one joint of which is restricted.
Using MPO, as example of offline RL method, we train a policy on dataset (D' U UD} and deploy the acquired policy in both source and target MDPs. As shown in Figure 2 (middle-top. w/o Aug.). ), such training paradigm does not produce an adaptive policy for the target. By modifying rewards in
7

---

Published as a conference paper at ICLR 2022
Table 1: Normalized scores for the (target) D4RL tasks, where our results are averaged over seeds.
The arrows in each four-tuple indicate whether the current performance has improved (f) or not (.) compared to the previous value. If 10+ DARA achieves comparable (less than 10% degradation) or better performance compared to baseline 10T, we highlight our scores in bold (in each four-tuple).
IT+10S w/o Aug.
IT+10S DARA
IT+10S w/o Aug.
IT+ DARA
IT+
IT+10S
Body Mass Shift
10T
IT
10T
10T
Aug.
DARA
BEAR
BRAC-p
AWR
Hopper
Random
11.4
11.0
10.9
9.6
11.0
10.2
10.3
3.4
4,5
Medium
52.1
0.8
0.9
1.6 1
32.7
29.0
29.2
32,9
35.9
30.9
20.8
28.9
Medium.R
18.2
34.1
20.1
28.4
8.8
4.1
4.2
Medium-B B
96.3
0.8 J
0.6
1.2 1
1.9
34.5
32.3
34.7
27.1
27.0
26.8
26.6
BCQ
CQL.
Hopper
Random
10.6
10.6 L
8.3
9.7 1 84.2
10.8
10.6 |
10.2. |
10.4
11.7
4.8
2.0
2.1
Medium
37.1
25.7
38.4
43.0
44.9
28.0
4.1
5.0
Medium-R
33.1
9.3 J
28.7
32.8
48.6
9.6
1.4
3.7
67.5
1.0
5.5
8.4
Modium-B
110.9
75.4
98.7
59.7
53.6
23.7
4.8"
5.8
BEAR
BRAC
AWR
Walker2d
Random
7.3
1.3
2.0
2.4
Medium
59.1
-0.5 1
0.6
0.3
77.5
6.4
70.0
78.0
17.4
14.8
17.1
17.2
Medium-R
7.3
18.6
15.5
7.4
1.6
1.5
Medium-B
40.1
-0.1 4.
1,5
2.3
76.9
20.6 J
64.1
77.5
53.8
35.5
52.5
53.3
CQL
MOPO
Walker2d
Random
4,9
1.8 L
4,5 +
4.8
7.0
1.7 1 49.5
3.2 1
3.4
13.6
-0.2
-0.1
-0.1
Medium
53.1
32.8 ]
50.9
52.3
79.2
42.9
80.0
81.7
17.8
7.0
5,7
11.0
Modium-R
15.0
6.9
14.9
15.1
26.7
4.6 1
0.8
2.0
39.0
5.1
3.1
14.2
Medium-B
57,5
32.5
55.2
57.2
111.0
63.5
93.3
44.6
5.3
5,5
17.2
source D', we show that applying the same training paradigm on the reward augmented data exhibits a positive transfer ability in Figure F (middle-bottom, DARA). In Figure 2 (right), we show that our DARA produces low Q-values on the obstructive state-action pairs (in left) compared to the vanilla MPO, which thus prevents the Q-value weighted-regreesionn on these unproductive state state-action pairs.
More generally, we illustrate how DARA can handle the dynamics adaptation from the reward mod- ification view. In Figure 3 (right), the learned reward modification -r (dashed blue line) clearly produces a penalty (red slices) on these state-action pairs (in source) that produce infeasible next- state transitions in the target MDP. If we directly apply prior offline RL methods, these transitions that are beyond reach in target and yet are high valued, would yield negative transfer. Thus, we can think of DARA as finding out these transitions that exhibit dynamics shifts and enabling dynamics adaptation with reward modifications, e.g., penalizing transitions covered by red slices (-Ar < 0).
6.2 CAN DARA ENABLE AN ADAPTIVE POLICY WITH REDUCED OFFLINE DATA IN TARGET?
To characterize the offline dynamics shift, we consider the Hopper, Walker2d and Halfcheetah from the Gym-MuJoCo environment, using offline samples from D4RL as our target offline dataset. For the source dataset, we change the body mass of agents add joint noise to the motion, and, similar to D4RL, collect the Random, Medium, Medium-R and Medium-E offline datasets for the three en- vironments. Based on various offline RL algorithms (BEAR, BRAC-p. BCQ, CQL, AWR, MOPO), we perform the following comparisons: 1) employing the 100% of D4RL data (10T), 2) employing only 10% of the D4RL data (IT'), 3) employing 10% of the D4RL data and 100% of our collected source offline data (IT+10S w/o Aug.), and 4) employing 10% of the D4RL data and 100% of our collected source offline data along with our reward augmentation (IT+I0S DARA). Due to page limit, here we focus on the dynamics shift concerning the body mass on Walker2d and Hopper. We refer the reader appendix for more experimental details, tasks, and more baselines (BC, COMBO).
As shown in Table I in most of the tasks, the performance degrades substantially when we decrease the amount of target offline data, i.e., 10T + IT. Training with additional ten times source offline data (/7+ //7+/0S w/o Aug.) also does not bring substantial improvement (compensating for the reduced data in target), which even degrades the performance in some tasks. We believe that such degradation (compared to 10T") is caused by the lack of target offline data as well as the dynamics shift (induced by the source data). Incorporating our reward augmentation, we observe that compared to /T and IT+10S w/o Aug. that both use 10% of the target offline data, IT+ IOS DARA significantly improves the performance across a majority of tasks. Moreover, DARA can achieve comparable or better performance compared to baseline 10T that training with ten times much target offline data.
8

---

Published as a conference paper at ICLR 2022
Table Normalized scores in (target) D4RL tasks, where "Tune" denotes baseline "fine-tune" We
observe that with same amount (10%) of target offline data, DARA greatly outperforms baselines.
Body Mass Shift
Tune
DARA
Tume
DARA
Tune
DARA
Tume
DARA
Tune
DARA
TyT
BEAR
BRAC-p
BCQ
CQL
MOPO
MABE
Hopper
Random
0.8
8.4
6.0
11.0 1
8.8
9.7
31,6
10.4 1
0.7
2.1 1
10.6
9.0
Medium
0.8
1.6
22.7
32,91
31.7
38.4
44.5
59.3
0.7
10.7
48.8
23.1
Medium-R
34.1
27.5
32.8
3.7
0.6
17.1
Medium-E
1.2
19.2
34.7
85.9
84.2
47.6
99.7
2.2
5.8
28.1
38.9
BEAR
BRAC-p
BCQ
CQL
MOPO
MABE
Walker2d
Random
3.2
3.9
3.2 4
4.7
3.4
0.1
-0.1 4
6.0
-0.2
Medium
0.3
0.3
76.0
78.0
28.4
52.3
72.3
81.7
-0.2
11.0
30.1
56.7
Medium-R RR
1.2
7.3
10.0
18.6
10.4
15.1
1,8
2.0
0.0
14.2
13.3
12.5
Medium-E
2.4
2.3
74.5
77.5 1
22.7
57.2
68.6
93.3
7.3
17.2
43.7
82.7
6.3 CAN DARA PERFORM BETTER THAN CROSS-DOMANT BASELINES?
In Section 6.2 IT+ w/o Aug. does not explicitly learn policy for the target dynamics, thus one proposal IOS fine-tune) for adapting the target dynamics is fine-tuning the model that learned with source offline data, using the (reduced) target offline data. Moreover, we also compare DARA with the recently proposed MABE [Cag et al. 202T), which is suitable well for our cross-dynamies setting by introducing behavioral priors Tp in the model- based offline setting. Thus, we implement two baselines, 1) IT+ IOS MABE upT and 2) 17+ IOS MABE Trp which denote 1) learning "p with target domain data and î with source domain data, and 2) learning Tp with source domain data and T with target domain data, respectively. We show the results for the Walker (with body mass shift) in Table 2 and more experiments in Appendix A.3.5 Our results show that DARA achieves significantly better performance than the naïve fine-tune-based approaches in a majority of tasks (67 "1" vs. "L" including results in appendix). On twelve out of the sixteen tasks (including results in appendix). DARA-based methods outperform the MABE-based methods. We attribute MABE's failure the difficulty of the reduced target offline data, which limits the generalization of the learned Tp or T under such data. However, such reduced data (10% of target) is sufficient to modify rewards in the source offline data, which thus encourages better performance for our DARA.
For real-world tasks, we also test DARA new offline dataset on the quadruped robot (see appendix for details). Note that can not access the privileged information (e.g., coordinate) in real robot, thus the target offline data (collected in real-world) does not contain rewards. This means that prior fine-tune-based and MABE- methods become unavailable. However, reward augmentation frees us from the requisite of rewards in target domain. We can freely perform offline training only using the augmented source offline data as long as the learned Ar is sufficient. For comparison, we also employ baseline (w/o Aug.): directly deploying the learned policy with source data into the (target) real-world. We present the results (deployed in real with obstructive stairs) in Table 3 and videos in supplementary material. We can observe that training with our reward augmentation, the performance can be substantially improved. Due to page limit, we refer readers to Appendix A.3.6 for more experimental results and discussion.
Table Average distance cov- ered in an episode in real robot.
(BCQ)
w/o Aug.
DARA
Medium Medium-E
0.85
1.35
1.15
1.41
Medium-R-F
1.27
1.55
7 CONCLUSION
In this paper, we formulate the dynamics shift in offline RL. Based on prior model- and model- free offline algorithms, we propose the dynamics-award reward augmentation (DARA) framework that characterizes constraints over state-action-next nextionn-ex--ttteee distributions. Empirically we demonstrate DARA can climinate the dynamics shift and outperform baselines in simulated and real- world tasks.
we characterize our dynamics-aware reward augmentation from the density reg- ularization view, which shows that it is straightforward to derive the reward modification built on prior regularized max-return objective e.g., AlgaeDICE (Nachum et al. 2019b). We list some re- lated works in Table 4 where the majority of the existing work focuses regularizing state-action distribution, while dynamics shift receives relatively little attention. Thus, we hope to shift the focus of the community towards analyzing how dynamics shift affects RI and how to eliminate the effect.
In Appendix A.2

---

Published as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
Our experimental evaluation conducted with publicly available D4RL (Fu et al., 2020) and Ne- oRL (Qin et al. 2021). In Appendix An4 A.5 we provide the environmental details and training setup for our real-world sim2real tasks. In supplementary material, we upload our source code and the collected offline dataset for the the quadruped robot.
ACKNOWLEDGMENTS
We thank Zifeng Zhuang, Yachen Kang and Qiangxing Tian for helpful feedback and discussions. This work is supported by NSFC General Program (62176215).
REFERENCES
Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Rémi Munos, Nicolas Heess, and Mar- tin A. Riedmiller. Maximum a posteriori policy optimisation. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Confer- ence Track Proceedings. OpenReview.net, 2018.
Philip J. Ball, Cong Lu, Jack Parker-Holder. and Stephen J. Roberts. Augmented world models facilitate zero-shot dynamics generalization from a single offline environment. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 619-629, PMLR, 2021.
Catherine Cang, Aravind Rajeswaran, Picter Abbeel, and Michael Laskin. Behavioral priors and dynamics models: Improving performance and domain transfer in offline RL. CoRR, abs/2106.091 19, 2021. URL BolLhtts://aaxit
Yevgen Chebotar, Karol Hausman, Yao Lu, Ted Xiao, Dmitry Kalashnikov, Jacob Varley, Alex Irpan, Benjamin Eysenbach, Ryan Julian, Chelsea Finn, and Sergey Levine. Actionable models: Unsupervised offline reinforcement learning of robotic skills. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18- 24 July 2021, Virtual Event, volume 139 of Proceedings Machine Learning Research, pp. 1518- 1528. PMLR, 2021. URL http: //tp:://prroe http::://pccceeedieeoo html
mlr www.r.press/vvli /ttaaa//l13//cheetartl.... chebotar2a........
Jinglin Chen and Nan Jiang. Information-theorete considerations in batch reinforcement learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICMI 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 1042-1051. PMLR, 2019.
Xinyue Chen, Zijian Zhou, Zheng Wang, Che Wang, Yanqiu Wu, and Keith Ross. Bail: Best-action imitation learning for batch deep reinforcement learning. arXiv preprint 19Xiv:1910.11 12179 2019.
Erwin Coumans and Yunfei Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. http::/pybbulett. org
Benjamin Eysenbach and Sergey Levine. If maxent RI. is the answer, what is the question? CoRR, abs/1910.01913, 2019. /tt.htttt////rrx....rr/r////////
01.0..1191 9193
Benjamin Eysenbach, Shreyas Chaudhari, Swapnil Asawa, Sergey Levine, and Ruslan Salakhutdi- nov. Off-dynamics reinforcement learning: Training for transfer with domain classifiers. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: datasets for deep data-driven reinforcement leamnin. CoRR, abs/2004.07219, 2020. URL https: /arxiv. org/abs 2004. 2004.07219
10

---

Published as a conference paper at ICLR 2022
Scott Fujimoto, David Meger, and Doina Precup. Off- off-policy deep reinforcement learning without exploration. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, Cali- fornia, USA, volume 97 of Proceedings of Machine Learning Research, pp. 2052-2062. PMLR, 2019.
Matthieu Geist, Bruno Scherrer, and Olivier Pietquin. A theory of regularized markov decision processes. Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, ICML 2019. 9-15 June 2019, Long Beach, Cali- fornia, USA, volume 97 of Proceedings of Machine Learning Research, pp. 2160-2169. PMM....RR 2019.
Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. Emaq: Expected- max q q-learning operator for simple yet effective offline and online RL. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 3682 3691. PMLR, 2021.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Picter Abbeel, and Sergey Levine. Soft actor-critic al- gorithms and applications. CoRR, abs/1812.05905, 2018. URL http: org/ /bs/ 1812. 05905|
Behzad Haghgoo, Allan Zhou, Archit Sharma, and Chelsea Finn. Discriminator augmented model- based reinforcement learning. CoRR, abs/2103. 129999 2021. URL httttts://aaiiv..ottt
abs/2103. 12999
Atil Iscen, Ken Caluwaerts, Jie Tan, Tingnan Zhang, Erwin Coumans, Vikas Sindhwani, and Vincent Vanhoucke. Policies modulating trajectory generators. In 2nd Annual Conference Robot Learning, CoRL 2018, Zürich, Switzerland, 29-31 October 2018, Proceedings, vol- ume 87 of Proceedings of Machine Learning Research, pp. 916-926. PMLR, 2018. URL http:://pptt httt:::/pproceddnggs... .mr...mrr..... wwwress/v8//ssee 18a.htm......mmmm.mm html
Nan Jiang and Jiawei Huang. Minimax value interval for off- evaluation and policy op- timization. Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tier Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL httpps::/ppoccce..gs....om lcdl 413204999560 4bb3844bbbbb hbw.ee/eebbbbbbeeeeeeeeeeee2eee Abst html
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model- based offfine reinforcement learning. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information P Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
Iya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In Marina Meila and Tong Zhang (eds.). Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 5774-5783 PMLR, 2021.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy g-learning via bootstrapping error reduction. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence Alché-Bus Buc, Emily B. Fox. and Roman Garnett (eds.), Advances in Neu- ral Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 11761-11771, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.), Advances in Neural Information Processing Systems 33: An- mual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020a.
11

---

Published as a conference paper at ICLR 2022
Saurabh Kumar, Aviral Kumar, Sergey Levine, and Chelsea Finn. One solution is not all you need: Few-shot extrapolation via structured maxent RL. Hugo Larochelle, Mare' Aurelio Ranzato. Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neu- ral Information Processing Systems 2020, NearIPS 2020, December 6-12, 2020, vir- wttppppppe//2///aass... Abstract. html
tual, 2020b. URL https:://proceeddngg new.eeeeeeinng....i.....s......
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforce- ment learning, pp. 45-73. Springer, 2012.
Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learn- ing quadrupedal locomotion over challenging terrain. Science Robotics, 5(47), 2020. doi: 10.1 http://ccrooootissaac55565 URL htttt:://robotiieeee http://bbbttccc....ceemeeeeer///
5/47/eabc5988
Sergey Levine. Reinforcement learning and control probabilisttc inference: Tutorial and review. CoRR, abs/1805.00909 2018. httt:h/tp//aaee
00909
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning:
Tutorial, review, and perspectives on open problems. CoRR, abs/2005.01643, 2020. URL
https: http:////aaaii.oo/aaaa//////////
Jinxin Liu, Hao Shen, Donglin Wang, Yachen Kang, and Qiangxing Tian. Unsupervised domain adaptation with dynamics-aware rewards in reinforcement learning. Advances Neural Informa- fion Processing Systems, 34, 2021.
Qiang Liu, Lihong Li, Ziyang Tang, and Dengyong Zhou. curse of
Breaking the
Infinite-horizon off-policy estimation. URL http:://prooeednng..eeeree Gbaoff86645000688443333--------3
horizon:
pp. 5361-5371, cc cc/paperr2018//aat
2018.
Eric Mitchell, Rafael Rafailov, Xue Bin Peng. Sergey Levine, and Chelsea Finn. Offline meta- reinforcement learning with advantage weighting. In Marina Meila and Tong Zhang (eds.), Pro- ceedings of the 38th International Conference Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 7780-7791. PMLR, 2021.
Ofir Nachum and Bo Dai. Reinforcement learning via fenchel-rockafellaa duality. CoRR, 155/2001.01866 2020. http:::://a....oo//////tt//
01666
Ofir Nachum, Yinlam Chow, Bo Dai. and Lihong Li. Dualdice: Behavior-agnoste estimation of discounted stationary distribution corrections. arXiv preprint arXiv: 04733, 2019a.
Ofrr Nachum, Bo Dai, llya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. AI- gaedice: Policy gradient from arbitrary experience. CoRR, abs/1912.02074, 2019b. URL
http: org/ http:/bs//11111 02074
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. CoRR, fbs/2000609359, 2020. URL http:s/////axxvv.rrr
abs/200s 09359
Sinno Jialin Pan and Qiang Yang. survey on transfer learning. IEEE Trans. Knowl. Data Eng., . 22 (10):1345-133) 2010. doi: 10.1 10.1109/KKDE.2009.199 191. URL https://dccci wttp:/i...rr.....1.11........... TKDE. 2009.2209..10
Xue Bin Peng, Marcin Andrychowicz. Wojciech Zaremba, and Pieter Abbeel. Sim-to-real trans- fer of robotic control with dynamics randomization. In 2018 IEEE International Conference on Robotics and Automation, ICRA 2018, Brisbane, Australia, May 21-25, 2018, pp. 1-8. IEEE, URL https: tttp:/too..g....1111 11099/I.......R....
2018. doi: 10.1109//C...211 2.486665.......
201 8. 8460528
12

---

Published as a conference paper at ICLR 2022
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Addantagg--eightee regression: Simple and scalable off off-policy reinforcement learning. CoRR, abs/1910.00177, 2019. URL
http:://rxiv org/ abs 111019900..0 11.990000000 00177
Jan Peters, Katharina Mulling, and Yasemin Altun. Relative entropy policy search. In Twenty-Fourth AAAI Conference on Artificial Intelligence, 2010.
Rongjun Qin, Songyi Gao, Xingyuan Zhang, Zhen Xu, Shengkai Huang, Zewen Li, Weinan Zhang, and Yang Yu. Neorl: A near real-world benchmark for offline reinforcement learning. arXiv 2021.
preprint arXiv:2102.000
Y. Sakakibara, Kan, Y. Hosoda, M. Hattori, and M. Fujie. Foot trajectory for a quadruped walking machine. In EEE International Workshop on Intelligent Robots and Systems, Towards New Frontier of Applications, pp. 315-322 vol.1, 1990. doi: 10.1 109//RO..1990.2624440 1990.2622077
Noah Y Siegel, Jost Tobias Springenberg, Felix Berkenkamp. Abbas Abdolmaleki, Michael Ne- unert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. arXiv preprint arXiv:2002.08339, 2020.
Richard S. Sutton. Integrated architectures for learning, planning, and reacting based on ap- proximating dynamic programming. In Bruce W. Porter and Raymond Mooney (eds.), Machine Learning. Proceedings of the Seventh International Conference on Machine Learn- ing, Austin, Texas, USA, June 21-23, 1990, pp. 216-224. Morgan Kaufmann, 1990. doi:
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba. and Pieter Abbeel. Do- main randomization for transferring deep neural networks from simulation to the real world. 2017.
Masatoshi Uchara, Jiawei Huang, and Nan Jiang. Minimax weight and f-function learning for off- policy evaluation. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 9659999599999 96666 PMLR, 2020. URL URLhttp: /ttp:////oceee/ngg..... mlr press/v119/ uehara20a. html
Nino Vieillard, Tadashi Kozuno, Bruno Scherrer, Olivier Pietquin, Rémi Munos, and Matthieu Geist. Leverage the average: an analysis of KL regularization in reinforcement learning. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neu- ral Information Processing Systems 2020, NeurlPS 2020, December 12, 2020, virtual, 2020.
Xingxing Wang. Unitree robotics.
2020.
Ziyu Wang. Alexander Novikov, Konrad Zolna, Jost Tobias Springenberg, Scott Reed, Bobak Shahriari, Noah Siegel, Josh Merel, Caglar Gulcehre, Nicolas Heess, et al. Critic regularized regression, arXiv preprint arXiv:2006.151: 2020.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforeement learning. CoRR, abs/1911.1 1.11661, 2019. URI.
11361
Haoran Xu, Xianyuan Zhan, and Xiangyu Zhu. Constraints penalized q-learning for safe offline reinforcement learning. CoRR, Abs/2107.09003. 2021. URL
2107.09005
Mengjiao Yang, Ofir Nachum, Bo Dai, Lihong Li, and Dale Schuurmans. Off-policy evaluation via the regularized lagrangian. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria- Florina Balcan, and Hsuan- Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurIPS 2020. De- cember 12, 2020, virtual, 2020. URL httpt:: URUtttpp:///occeedin htt.:pp:s/ppreeeccnn...ee.....ss nwwrips...p..... cttpppppcccppppe//
13

---

Published as a conference paper at ICLR 2022
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y. Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: model- based offiiee policy optimization. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell. Maria-Florina Balcan, and Hsuan- Tien Lin in Neural Information Processing Systems 33: Annual Conference on Neural Information Pro- cessing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. COMBO: conservative offline model based policy optimization. CoRR, abs/2102.08363, 2021. URL htttss://axxv..rrrt org/ ab1/ 08363
Hongyin Zhang, Jilong Wang. Zhengqing Wu, Yinuo Wang, and Donglin Wang. Terrain-aware maassessmentt-ntwork..
deep reinforcement learning for quadrupedal locomotion in tough terrain. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4538 4545. IEBR.
Ruiyi Zhang. Bo Dai, Lihong Li, and Dale Schuurmans. Gendice: Generalized offline esti- mation of stationary values. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, OpenReview.net, 2020. URL https: //openrevi ew, net/forum's d=Hkxl cnVFwB
14

---

Published as a conference paper at ICLR 2022
A APPENDIX
A.1 DERIVATION
A.1.1 PROOF OF LEMMA
Let BT (s) (nws(a (r(s.a) a) + = [V(']]] and r(s) En-r(aas) (r(s. a)). Then,
we have
Fa (T) - 77d (T) = Exompo(s) (So) - Vas (so)
E P n. n, By Barm(a (r(se, at)l Esompo(s) [Va()))]
"B B P MM (r(s)) + Vss(s:) Var(s))] Beonpu(s) [M)
t=0
E 8/ Pla. a, M0 P(e.lllM
[r(s:) + +Var(54+1) Var(sr)
'1 P(s M t=0
[r(s:) + (VMM((L))) (r(s:) + 7 Ba Vvs (s)))
P M (+11)
? E P P(s n. .1)) B VM (s:) B V.r (s:)
t=0
Byod
B M (s) BM Vas
A.1.2 MODEL-BASED FORMULATION
Here we provide detailed derivation of the lower bound in Equation in the main text.
Assumption Assume scale c and a function class F such that VM E cF.
Following MOPO (Yu et al. 2020), we set F {f |l I 1}. In Section Preliminaries, have that the reward function is bounded: r(s, a) E [--Rmaas Rmaa]. Thus, we have || S
Ei Rman= 1-7
Rmus and hence the scale Pance 1-7
As direct corollary of Assumption ]and Equation 5 we have
Faaa (8) [BT Mr. VM((s) - B% (s)] S 7c R Bg (s)a(als) [d>(T(s''ss (s'(ss a)), T(s|s, a))]
(10)
Further, we find
(s'(ss a), ((sls a)) S dz a), (s'ss,)) + dx (T a), ((s'ss)))
(11)
For the first term dr (1"
(s' a)) in Equation 12 through Pinsker's inequality, we have
dz (T' (s'1s, a), (s'ss, a)) = Drv (s' a), (s'ls,
DrL. DRa.(T (T' a), (s' a))
(12)
To keep consistent with the DARA- baseed method-free free offline methods, introduce scale 7 and
bias climinate the square root Equation 12 To be specific, we assumd scale 7 and bias
such that v DKL (T', 7) $ n Dk. (T',
(",T) 7) + 5. Thus, we obtain
a)) + dx a). T('s a)) Drv 7 I nDKL (s'(s, (ss ss))s
(13)
In implementation. we clip maximum deviation log ((r
('(a'aa.) for each (s, ass).. which thus makes
DRL DRI(T'S (T'(s's,, (s''s,, a): (ss"|s, a)) bounded.
15

---

Published as a conference paper at ICLR 2022
For the second term dy (T'|ss, a), T(''ss s. a)) in Equation 11 oracle uncertainty qualification module that provides an upper bound on the error of the estimated empirical MDP M == Ar Po. 70
assume that have access
Assumption Let be the function class in Assumption [ We say S A R is admissible error estimator for T dy (s'sss a), ((s')s, a)) S u(s, a) for all (s, a).
Thus, we have
aa (s)s)))s) Me
(T'(Tss ((s)ss, a)) S Bg (s)a(ais) (m(s,a))
(14)
Bring Inequations 10
13 and
14
into Lemma
we thus have
r(s, 77clog ('s'ss, -o) -00 (15)
77M (TT) 2 E Ra " (1)())
708
T(s'ss
A.1.3 LEARNING CLASSIFIERS
Applying Bayes' rule, we have
p(source|s (s, s' s') s's
('('aa, =- p(s'|s, as source)
p(source a a) a)
T(s'la,s) p(s'|ss.a target) p(target|s, s')p(ss s')
p(target)s. a)
Then we parameterize p(-|s, a, s') and p(- a) with the two classifiers asas and Qsa respectively, Us- ing the standard cross-entropy loss, we learn Ias and Qa with the following optimization objective:
(s...s"))~ [log Yas (source|s, a, s')] + (ss.,.5 [log (an (target|s, a, s,))),
max
Ess.a)=Dr [log 9sal (source a)] (s.a)>D [log Qsa (targett) a)].
With the trained Asss and gsa. have
(s'ss, Isns (source|s a's') 2sa 2s a) log (s)
llog
loog Isas (target|s s') Isa (tagettt|| a)
(16)
In our implementation, we also clip the above reward modification between - 10 and 10.
A.2 REGULARIZATION VIEW OF DYNAMICS-AWARE REWARD AUGMENTATION
Here we shortly characterize our dynamics-awerr aware reward augmentation from the density regulariza- tion. Note the standard max-retum objective nM (TT) in RL can be written exclusively in terms of the on-policy distribution d (s) s). To introduce an off-policy distribution dp (s (s) in the ob- jective, prior works often incorporate regularization (penalty): D(d (s) s)]ld (s) (als))) as in Equations and However, facing dynamics shift, such regularization should take into account the transition dynamics, which is penalizing (dd,, (a)s( (alas]([ (s' (s. d (S) (a s) {)) (s'1s a)). From this view, it is also straightforward derive the reward modification built on prior regularized off-policy max-return objective e.g., the off- off-policy approach AlgaeDICE (Nachum et al. 2019b).
In Table we provide some related works with respect the (state-action pair) dp (s) (a]s) regularization and the (state-action nextoon--ett-tttee pair) d' (s)TT (als)?" a) regularization. We can find that the majority of the existing work focuses regularizing state-action distribution, while dynamics shift receives relatively little attention. Thus, we hope to shift the focus of the community towards analyzing how the dynamics shift affects RI. and how eliminate the effect.
16

---

Published as a conference paper at ICLR 2022
Table Some related works with explicit (state-action p(s, a) or state-action-next-steee p(s, a, s')) regularization. More papers with respect to unsupervised RL, inverse RL (imitation learning), meta RL, multi-agent RI., and hierarchical RL are not included.
reg. with do (8) Ta (a(s)
reg. with dp (s) (s
Online:
see summarization in Geist et al.
Eysenbach et al. (021 (DARC)
2019) and Vieillard al. (2020).
Liu al. [2021
(DARS);
Haghgoo et al.
2021
Offtine (off-policy evaluation):
Fujimoto et al.
(2019) (BCQ);
Kumar et
(2199)
(BEAR);
Wu al. 2019 (BRAC-p);
Abdolmaleki et al.
(018)) (MPO);
Peng et al.
20T9
(AWR);
Nair et al.
[2020] (AWAC);
Wang et al.
2020
(CRR);
Siegel et al. (2020);
Chen et al.
(2019); (BAIL)
Kumar et al.
(2020a)
(CQL);
Xu ct al.
2021 (CPQ);
Kostrikov ct al. (2021) (Fisher-BRC);
Liu ct al.
2018);
Nachum ct al.
2019a (DualDICE);
Nachum al. (201bb) (AlgaeDICE);
Zhang al. (2020) (GenDIIE))
Yang et al. (2020);
Nachum & Dai (2020);
Jiang & Huang (2020)
Uehara al. (2020):
Yu et al.
22020
(MOPO):
Kidambı et al. (2020) (MOReL);
Yu (202T)
(COMBO);
Cang et
A.3
MORE EXPERIMENTS
A.3.1 TRAINING WITH (D'UD)
As we show in Figure iin Section Introduction, the performance of prior offline RL methods dete- riorates dramatically the amount of (target) offline data D decreases. In Figure we show that directly training with the mixed dataset {D' U D) will not compensate for the deteriorated perfor- mance caused by the reduced target offline data, and training with such additional source offline data can even lead the performance degradation in some tasks.
Mediuun-Rspaay
Mediumn-Rppay
Mediinn-Epeot
Mediten-Epeel
NonaToo seaee
Nonstedae
20
30%
10P%
19%
5%
50%
10%
10%
5%
52%
20%
12%
5%
50%
20%
13%
5%
Assomnt of Laguet datu andd traning
Assount of lagas dats aned fo traning
Assrai of Lages Jata used for tniiiin
Ausual of lagas data usd fartraitt
CCC Servce
COC.
MOPO Seurcs
MOTO
CXg with Sauce
MIDP Soucce
MOPO
Figure 4: Final performance on the D4RL (Walker2d) task: The orange bars denote the final per- formance with different amount (50%D, 20%D. 10%D, 5%D) of target offline data; The blue bars denote the final performance of mixing 100% of source offline data D' and different amount of target data x%D (x E [50, 20, 10, 5)). i.e., training with (100%D' U x%D); The red lines denote the final performance of training with 100% of target offline data D. We can observe that 1) the performance deteriorates dramatically as the amount of (target) offline data decreases (100%D (red line) > 50%D (orange bar) 20%D (orange bar) - 10%D (orange bar) 5%D (orange bar)), 2) after training with the additional 100% of source offline data, {100%D' U x%D). the final performance improved in some tasks, but most of the improvement is a pittance compared the original per- formance degradation (compared to that training with the 100% of target offline data, i.e., the red lines), and 3) what is worse is that adding source offline data D' even leads performance degradation in some tasks, e.g., CQL with 50%D D and 20%D in Mediumm-Randon
17

---

Published as a conference paper at ICLR 2022
A.3.2
COMPARISON BETWEEN LEARNING CLASSIFIRSS AND LEARNING DYNAMICS (FOR THE REWARD MODIFICATION)
Table Normalized scores for the Hopper tasks with the body mass (dynamics) shift. Rat. and Cla. denote estimating the reward modification with the estimated-dynamies ratio and learned classifiers (Appendix A.1.3 respectively.
Body Mass Shift
BEAR
BRAC-P P
AWR
BCQ
CQL.
MOPO
Cla.
Cla.
Cla.
Cla.
Happer
Random
9.9
8.4
11.2
11.0
3.7
4.5
8.5
9.7
11.8
10.4
1.8
2.1
Medium
1.6
31.7
32.9
18.0
28.9
33.2
38.4
45.9
59.3
3.1
10.7
Medium-R
28.4
34,1
36,5
30,8
2.5
4.2
33,9
32.8
2.0
3.7
3.8
8.4
Medium-E
1.2
50.9
34.7
45.8
26.6
68.4
84.2
107.3
99.7
5.7
5.8
In Table 5 we show the comparison between learning classifiers and learning dynamics (for our reward modification) in the Hopper tasks. We observe that the schemes for estimating the reward modification have similar performance. Thus, for simplicity and following Eysenbach et al. adopt the classifiers modify rewards in the source offline data in our experiments.
202T.
A.3.3
MORE EXAMPLES WITH RESPECT TO THE REWARD AUGMENTATION
MIN
MDUM
0.26 0.26
Statf
-0.6 0.26
State Il]
-0.26 0.26
Time step
Time step
Time
Figure 5: We observe that reward augmentation 1) encourages A-Ar O, Le., the green
slice parts) these transitions (-0.26 < next-state[11] S 0.26) that have the same dynamics with the target environment, and 2) discourages (-Ar < Ô, i.e., the red slice parts) these transitions that have different (unreachable) dynamics (next-state[1] -0.26 or soxt-state[1] 2 0.26) in the target.
In Figure 5 we provide more examples with respect to the reward augmentation in the Ant task in Figure (left).
A.3.4 COMPARISON BETWEEN 1OT, IT, IT+ IOS w/o Aug., AND /7+ IOS DARA
Based on various offline RL algorithms (BEAR (Kumar et al. 2019) BRAC-P (Wu et all 2019 BCQ (Fujimoto al. 2019). CQL (Kumar et al. 2020a), AWR (Peng al. 2019), MOPO YYu et al. 2020), BC (behavior cloning), COMBO Yu et al. 2021 ), we provide the additional results in Tables
and 10
Table 6: Normalized scores for the Hopper tasks with the body mass (dynamics) shift. (The com- parison results for BEAR, BRAC-p, AWR, CQL, and MOPO are provided in the main text.)
IT+
IT+10S
IO
IT DARA
IT+10S
IT+10S
Body Shift
w/o Aug.
DARA
Aug.
w/o Aug.
DARA
BC
COMBO
Random
1
10.1
17.9
0.7
1
Medium
29.0
27.9
17.6
25.0
94.9
1.8
33.7 1
45.7
Medium-R
11.8
7.8
11.6
73.1
13.1
11.0 ]
27.9
Medium-E
111.9
21.5
20.8
35.7
111.1
0.8
14.9
108.1
18

---

Published as a conference paper at ICLR 2022
Table 7: Normalized scores for the Hopper tasks with the joint noise (dynamics) shift.
IT+10S w/o Aug.
IT+10S w/o Aug.
Joint Noise Shift
10T
IT
DARA
10T
IT
DARA
1GT
IT
wio Aug.
DARA
BRAC-p
Hopper
Random
11.4
0.6
7.4
4.2. |
11.0
10.3
10.0
10.8
10.2
10.1
3.6
4.0
Medium
27.6
Mediam-R
33.7
2.7
3.6
9.9
0.6
13.4
89.9
101.4
28.4
12.4
6.7
7.2
Medium-B
19.8
57.6
87.8
27.1
27.01
27.0
BOQ
CQL
MOPO
Hopper
11.7
1.5
1.3
Mediam
54,5
45.88
49.0
54.4
58.0
46.2
$8.0
58.0
28.0
2.7
9.2
17.3
Medium-R
33.1
13.0
23.8
32.0
48.6
13.6
2.6
3.6
67.5
0.8
2.3
6.4
Medium-F
110.9
44.6 )
96
109
98.7
50.7 1
73.4
108.9
23.7
14
6.1
7.5
BC
COMRO
Hopper
Random
9.8
9.8 7
7.5
9.17
17.9
0.7 1
1.8
4,9
Medium
29.0
27.9
29.0
29.0
94.9
9.6
Medium-F R
11.8
7.8
8.5
11.3
73.1
13.1 1
4.0
9.6
Medium-E
111.9
21.5
53.5
77.9
111.1
0.8
34.0
45.9
Table 8: Normalized scores for the Walker2d tasks with the body mass (dynamics) shift. (The comparison results for BEAR, BRAC-p, AWR, CQL, and MOPO are provided in the main text.)
IT+10S wlo Aug.
IT+10S DARA
IT+10S w/o Aug.
IT+ DARA
IT+ w/o Aug.
IT+10S DARA
Body Mass Shift
10T
10T
COMBO
Random
1.6
0.1
1.7
2.7
7.0
1.8 J
2.0
3,5
Walkera
Medium
6.6
5.5 L
3.8
6.6
75.5
-1.0
23.9
36.6 1
Medium
56.0
0.1
22.6
Medium-E
6.4
3.1
6.0
6.2
96.1
.0.9
-0.1
-0.1
Table 9: Normalized scores for the Walker2d tasks with the joint noise (dynamics) shift.
IT+ w/o Aug.
IT+10S DARA
IT+
IT+10S
Joint Noise Shift
10T
IT
DARA
10T
IT
w/o Aug.
10T
IT
w/o Aug.
DARA
BRAC-P
Walker2d
Random
7.3
2.2
0.6
2.6
-0.2
2.8
3.3
8.8
1.5
0.9
1.5
1.5
Medium
59.1
-0.4
0.6
0.1
77.5
28.8
55.2
72.9
17.4
12.2
17.2
17.2
Medium-R
19.2
0.4
10.4
-0.3
6.3
32.1
34.8
15.5
1.4
2.1
Mediumn-E
40.1
.0.2
0.8
0.6
76.9
21.8
62.3
74.3
53.8
40.4 ]
53
53.6
BCQ
CQL
MOPO
Walker2d
Random
4.9
3.7
3.4
5.2
7
0.5
2.7
6.4
13.6
-0.3
-0.2
-0.2
Mediumn
Medium-R
15
5.7
9.8
14.6
26.7
1.8
1.4
1.8
39
0.8
9.3
16.4
Mediumn
BC
COMBO
Walkerad
Randon
1.6
0.9
1.6
7.0
1.8
0.1
Modium
6.6
5.5
6.4
6.5
75.5
-1.0
0.4
0.7
Mediumn-
11.3
10.4
56.0
0.1
5.6
7.4
Modium-E
6.4
3.1
6.2
6.4
96.1
-0.9
0.8
-0.1
19

---

Published as conference paper at ICLR 2022
Table 10: Normalized scores for the Halfcheetah tasks with the joint noise (dynamics) shift.
IT+10S w/o Aug.
IT+10S DARA
IT+10S
Joint Noise Shift
10T
IT
DARA
10T
IT
w/o Ang.
10T
IT
w/o Aug.
DARA
BEAR
BRAC-P
AWR
Haltcheethh
Random
25.1
17.8
25.01
25.1
24.1
10.0
25.0
26.7
2.5
2.7
3.1
48.9
Mediumn
41.7
-0.2
0.8
1.5
43.8
43.0
52.4
53.0
37.4
38.2
48.7
37.4
Mediuum
38.6
-0.6 J
-0.5
45.4
2.5
-2.3
45.3
40.3
2.6
2.3
2.3
Medium-E
53.4
-1.2
1.0
-1.4
44.2
6.9
0.9
45.3
52.7
32.2 J
80.6
79.2
BCQ
CQL
MOPO
Halfchedtah Haltcheetah
Random
2.2
2.3
2.2
2.3
35.4
-2.3
-2.4
10.4
35.4
2.3
1.2
1.1
Mediumn
37.6
Medium-R R
38.2
1.1
39.4
41.3
46.2
0.6
2.0
1,9
53.1
-0.1
2.6
4.2
Medium
BC
COMBO
Random
4
2.2
2.2
38.8
24.0
18.7
20.3
Medium
36.1
36.5
49.4
49.8
54.2
15.7
14.9
15,9
Medium-R
38.4
36.5
24.6
15.7
55.1
-2.6
-2.4
4,8
Mediumm
35.8
36.3
49.0
49.3
90.0
4.4
6.5
11.1
A.3.5 COMPARISON WITH THE CROSS-DOMAIE BASED BASELINES
In Tables πI and 12 we provide the comparison between our DARA-based methods, fine-tune based methods, and MABE-based methods Hopper and Walker2d tasks, over the dynamics shift concerning the joint noise of motion. We can observe that in a majority of tasks, our DARA- based methods outperforms the tunetuue- method (67 "*" vs. 13 "4", including the results in the main text). Moreover, our DARA can achieve comparable or better performance compared to MABE-based baselines on eleven out of sixteen tasks (including the results the main text).
Table 11: Normalized scores the (target) D4RL Hopper tasks with the joint noise shift., where "Tune" denotes baseline "fine-tune".
Joint Noise Shift
Tune
DARA
Time
DARA
Tune
DARA
Tune
DARA
Tume
DARA
JpT
TTp
BEAR
BRAC-p
BCQ
CQL
MOPO
MABE
Hopper
Random
0.8
4,2
6.4
10.8
8.1
9.6
32.2
10.8
0.6
2.9
10.8
8.1
Medium
1.9
2.0
44.9
37.6
54.4
58.0
17.3
57.7
Medium-R
0.7
9.9
32.4
101.4
29.6
32.0
1.3
3.6
21.5
35.4
Medium-E
0.8
1.4
98.2
87.8
90.5
109.0
107.3
108.9
4.9
7.5
15.5
104.8
Table 12: Normalized scores in the (target) D4RL Walker2d tasks with the joint noise shift., where "Tune" denotes baseline "fine-tune".
Joint Noise Shift
Tune
DARA
Tune
DARA
Tune
DARA
Tune
DARA
Tune
DARA
F
TTP
BEAR
BRAC-p
BCQ
CQL
MOPO
MABE
Walker2d
Random
2.7
2.6
1.4
8,8
3.0
5.2
6.7
6.4
-0.4
-0.2
5.0
-0.2
Medium
0.5
0.1
55.8
72.9
45.1
52.7
76.6
81.2
7.0
12.2 1
49.4
48.7
Medium-R
3.2
10.4
12.2
34.8
13.5
14.6
-0.4
1.8
1.9
16.4
4.5
Medium-E
-0.4
71.7
74.3
104
116.5
84.7
82.6
A.3.6 ADDITIONAL RESULTS ON THE QUADRUPED ROBOT
In this offline sim2real setting, we collect the source offline data in the simulator (10% or * 10c steps) and target offline data in the real world (3 10 steps). See Appendix A.4 for details. For testing, we directly deploy the learned policy in the real (flat or obstructive) environment and adopt the average distance covered in episode (300 steps)
evaluation metrics.
20

---

Published as a conference paper at ICLR 2022
Figure 6: Illustration of the real environment (for testing): (left) the flat and static environment, (right) the obstructive and dynamic environment.
Table 13: Average distance (m) covered in an episode (300 steps) flat and static (real) environment.
Sim2real (Flat and Static)
w/o Aug.
DARA
w/o Aug.
DARA
w/o Aug.
DARA
BCQ
CQL
MOPO
Medium
1.56
1.64
1.80
1.82 1
0.00
0.00
Quadruped Robot
Medium-R
0.00
0.00
Medium-E
2.16
2.47
2.03
2.02 4
0.00
0.00
Medium-R-B
1.69
2.28
0.00
0.00
Average performance improvement
13.6%
0.2%
0.0%
(Flat and static environment) We first deploy our learned policy in the flat and static environment. The results (distance covered in an episode) are provided in Table
We find that with Medium-R offline data, w/o Aug. BCQ and DARA BCQ both could acquire the locomotion skills, which think is caused by the lack of high-quality offline data. With more "expert" data (Medium-R - Medium - Medium-E E, or Medium-R - Medium- R-E), w/o-Aug. BCQ allows for progressive performance (0.00 + 1.56 + 2.16, or 0.00 - 1.69 in BCQ), but with our reward augmentation, such performance can be further improved (with average improvement 13.6%).
1) BCQ (Figure 7
2) CQL (Figure 8): We find that with Medium-R or Medium-R-F R-E offline data, w/o Aug. CQL and DARA COL both could not learn the locomotion skills, which we think caused by the low-quality "Replay" offline data. With Medium or Medium-E offline data, w/o Aug. CQL and DARA CQL acquire similar performance on this flat and static environment.
3) MOPO: We find that the model- MOPO (both w/o Aug, and DARA) could hardly learn the locomotion skill under the provided offline data.
Table 14: Average distance (m) covered in an episode (300 steps) in the obstructive and dynamic (real) environment.
Sim2real (Ohstructive and Dynamic)
w/o Aug.
DARA
w/o Aug.
DARA
w/o Aug.
DARA
BCQ
CQL
MOPO
Medium
0.85
1.35 f
0.92
1.40
Quadruped Robot
Medium-R
Medium-E
1.15
1.41 1
0.77
1.32 1
Medium-R-E
1.27
1.55 1
Average performance improvement
25.9%
30.9%
(Obstructive and dynamic environment) We then deploy our learned policy in the obstructive and dynamic environment. The results (distance covered in an episode) are provided Table [4]
1) BCQ (Figure 9): In this obstructive environment, environment. With more "expert" data (Medium - Medium-E - Medium-BR R-E), w/o Aug. BCQ allows for progressive performance (0.85 > 1.15 - 1.27), and with our reward augmentation, such performance can further improved (with average improvement 25.9%). At the same time, also find that due the presence of environmental obstacles, the performance of both w/o
can obtain similar results as in the flat
21

---

Published as a conference paper at ICLR 2022
Aug. BCQ and w/o Aug. DARA is decreased compared to the deployment the flat environment. However, we find that our DARA exhibits greater average performance improvement under this obstructive environment (13.6% 25.9%) compared to that in the flat environment. These results demonstrate that our DARA can learn an adaptive policy for the targel environment and thus show greater advantage over w/o-Aug. in complex environments.
2) CQL (Figure TOp: Similar to BCQ, our DARA CQL exhibits a greater performance improvement over baseline in the obstructive and dynamic environment (0.2% 30.9% compared that in the flat and static environment.
In summary, The results in the quadruped robot tasks support conclusion in the main text regarding the dynamics shift problem in offline RL - with only modest amounts of target offline data (3 10 steps). DARA-based methods can acquire an adaptive policy for the (both flat and obstructive) target environment and exhibit better performance compared to baselines under the dynamics adaptation setting.
Medium (w/o Aug.)
1.56 m
Medium (DARA)
1.64 m
Medium- (w/o )
0.00 m
Medium-R R (DARA)
0.00
Medium-E (wlo Aug.)
2.16
Medium-E E (DARA)
2.47
Medium-R-H (w/o Aug.)
1.69 m
Medium-R-E (DARA)
m
Figure 7: Deployment on the flat and static environment of BCQ.
Medium (w/o Aug.)
1.80m
Medium (DARA)
1.82 m
Medium-E (w/o Aug.)
2.03 m
Medium-F (DARA)
2.02
Medium-R-4 (w/o Aug.)
0.00
Medium-R R-F - (DARA)
0,00 m
Figure 8: Deployment the flat and static environment of CQL.
22

---

Published as a conference paper at ICLR 2022
Medium (w/o Aug.)
0.85
Medium (DARA)
1.35
Medium-E (w/o Aug.)
1.15
Medium-E (DARA)
1.41
Medium-R-E (w/o Aug.)
1.27 m
Medium-Re R-E (DARA)
1.55
Figure 9: Deployment on the obstructive and dynamic environment of BCQ.
Medium (w/o Aug.)
0.92 m
Medium (DARA)
1.40
Mcdium-E E (w/o Aug.)
0.77 m
Medium-E (DARA)
1.32
Figure 10: Deployment on the obstructive and dynamic environment of CQL.
A.3.7 ABLATION STUDY WITH RESPECT TO THE AMOUNT OF TARGET OFFLINE DATA
To see whether the amount of target offline data can further reduced, we show the results of the ablation study with respect to the amount of target offline data in Tables BS and 16
Table 15: Ablation study with respect the amount of target Hopper data (body mass shift tasks). 10%, 5% and 1% denote training with 10%, 5% and 1% of target offline data, respectively, and additional 100% source offline data.
Body Mass Shift
10%
5%
1%
10%
5%
1%
10%
5%
1%
10%
5%
1%
BEAR
BRAC-p
BCQ
CQL
Hopper
Medium-R
34.1
10.7
30.8
27.7
20.0
32.8
20.5
2,3
Medium-F.
1.2
0.6
0.6
34.7
25.1
20.6
84.2
65.1
55.6
99.7
52.3
38.5
Table 16: Ablation study with respect the amount of target Walker2d data (body mass shift tasks). 10%, 5% and 1% denote training with 10%, 5% and 1% of target offline data, respectively, and additional 100% source offline data.
Body Mass Shift
10%
5%
1%
10%
5%
1%
10%
5%
1%
10%
5%
1%
BEAR
BRAC-p
BCQ
CQL
Walker2d
Medium-R
7.3
5.9
1.3
18.6
21.6
15.8
15.1
12.7
9.7
2.0
1.3
0.5
Medium-F
2.3
-0.2
-0.3
77.5
2,0
-0.2
57.2
29.7
20.6
93.3
0.1
-0.3
A.3.8 ILLUSTRATION OF WHETHER THE LEARNED POLICY IS LIMITED TO THE SOURCE OFFLINE DATA
we directly perform DARA with only the source offline data D' the learned behaviors will be restricted to the source offline data. For example, in the Map task, collecting source dataset with the obstacle and collecting target dataset without the obstacle. this case, can be harder for DARA (with only the source D') to capture the change in the transition dynamics, thus harder for the agent
23

---

Published as a conference paper at ICLR 2022
to figure out the new optimal policy (the shorter path without the obstacle). However, as stated in Algorithm| we perform offline RL algorithms with both target offline data and source offline data {D' U D). Thus, to some extent, such limitation can be overcome as long offline RL algorithm captures the information (eg. the short path without the obstacle) contained in the (limited) target D, see Figure
[7 for the illustration.
targel
target
target
DARA
DARA
DARA
DARA
DARA
Ok
Ik
2k
k
10
Figure 11: We exchange the source environment and the target environment in Figure (in the main text) so that the source environment has an obstacle and the target environment has no obstacles. In the source domain, we collect 100k of random transitions. the target domain, we collect Ok, 1k, 2k, 5k, and 10k random transitions respectively. We set 0.1. We can find that if we perform DARA with only source offline data D' (f.e., target data), we indeed can acquire the optimal trajectory (eg. the short path without the obstacle). However, even there is transition of passing through obstacles the source data, performing DARA with (D' D) enables us to acquire behavior of moving through obstacles. As we increase the number of target offline data D, training with (D' U D can gradually acquire optimal trajectories.
A.3.9 COMPARISON BETWEEN DARA AND IMPORTANCE SAMPLING (IS) BASED DYNAMICS CORRECTION
In T7] we report the experimental comparison between DARA and importance sampling based dynamics adaption. We can find that in most of the tasks, our DARA performs better than the IS- based approaches.
Table 17: Comparison between DARA and importance sampling (IS) based dynamics correction.
Body Mass Shift
IS
DARA
IS
DARA IS
DARA
BEAR
BRAC-P
AWR
Hopper
Random
4.6 2.8
8.4 + 1.2
10.8 0.5
11 1 0.6
10.2 + 0.3
4.5 0.9
Medium
1
1.6
17.4 10.6
32.9 7.5
24.8 7.7
5.5
Medium-R
17.3
34.1 5.8
21.6
30.8 4.9
142
4.2 3.5
Medium-E
0.8 + 0.2
1.2 0.5
36 13.5
8.5
29.3 2.6
26.6 22
BCQ
CQL
MOPO
Hopper
Random
9.2 + 1.1
9.7 + 0.2
10.3 + 0.4
10.4 + 0.4
2.8 + 3
2.1 + 1.7
Medium
38.4
+ 1.8
43.3
59.3 + 12.2
7.6
7.2
10.7 + 5.1
Medium-R
14.2 + 1.3
32.8
0.9
2.2 0.3
3.7 + 1.4
4.9
+
3.8
8.4 3.5
Medium-E
84.2 + 9.8
87.8 + 16.9
99.7 + 16.4
4.6
2.9
5.8 2.3
A. THE SENSITIVITY OF THE COFFFICIENT OF THE REWARD MODIFICATION
In Table 18, We check the sensitivity of parameeer 17, i.e., the coefficient of the reward modi- fication in (s. a) mAr(s,
24

---

Published as a conference paper at ICLR 2022
Table 18: We show the normalized scores for the Hopper tasks with body mass shift, by varying
n E {0, 0.05, 0.1, 0.5) over BEAR, BRAC-p, AWR, BCQ. CQL, and MOPO.
Hyper-parameter
Body Mass Shift
0.05
0.1
0.2
0.5
BEAR
Random
4.6 + 3.4
7.7 0.9
8.4 + 1.2
7 1.2
4.2 1.1
Hopper
Medium
0,9 0.3
0.6
1.6 +
0.7 0.1
18.2 Medium-R
28.5
34.1 5.8
29.1 4.4
18.1
Medium-E
0.6 + 0
0.8 0.1
1.2 = 0.5
1.2
0.7 0.1
BRAC-p
Random
9.6
11.2 = 0.8
11 0.6
10.6 2.4
5.3 1.2
Hopper
Medium
29.2 2
26.5 = 1.8
32.9 7.5
16.1 0.9
16.7 + 1.7
Medium-R
20.1 + 4.8
17.8 + 3.2
30.8 4.9
13.9
10.4 + 2.4
2.4
Medium-F
32.3 7.8
40.4 + 4.4
34.7 + 8.5
29.4 6.5
25.2 + 4.1
AWR
Random
3,4 + 0.7
4.1 +1
4.5 0.9
3.7
2.5 0.1
Hopper
Medium
20.8 = 6.3
2.9
28.9 5.5
2..6
1.5
4.1 1.7 Medium-R
3
3.5
0.6
4.3 1.3
Medium-E
26.8 = 0.4
27
26.6 2
17.8
24.2 3.9
BCQ
Random
0.3
9.6
9.7
7.4
7.6
Hopper
Medium
25.7 - 5.5
2.
38.4 = 1.8
27.1 1.7
26.7 0.8
Medium-R
28.7
1.9
29.5 3
32.8 0.9
25.9 4
21 2.2
Medium-F
75.4 + 7.8
70.4 + 5.4
84.2 + 9.8
67.9 8
61.9 4.3
CQL
Random
10.2 + 0.3
10 +
10.4 + 0.4
10 + 0
10 +
Hopper
Medium
4
59.8 6
59.3 + 12.2
44.2 11
37.1 2.6
Medium-R
1.4 1.4
0.2
3.7 : 1.4
3,9 1.7
3.4
Medium-E E
53.6 21.2
65.3 + 15.4
99.7 + 16.4
60.5 10.5111 16
75.9 + 30
MОPO
Random
2.1
1.8 0
2.1 1.7
1.2
0.8 - 0
Hopper
Medium
5
6.5
10.7 5.1
5.3 1.6
2.8 0.7
Medium-R
5.5 + 4.6
7.5 15 0.8
3.5
5.7 3.5
1.9 0.6
Medium-E
4.8 4 2.9
8.1 +
5.8 4.4+2.
4.7 + 0.7
2.1 0.2
25

---

Published as a conference paper at ICLR 2022
A.4 ENVIRONMENTS AND DATASET
Figure 12: Illustration of the suite of tasks considered in this work: (from left right) Hopper, Walker2d, Halfcheetah, simulated and real-world quadruped robots. These tasks require the RL. agent learn locomotion gaits for the illustrated characters.
In this work, the tasks include Hopper, Walker2d, HalfCheetah, simulated (see the dynamics param- eters /Zhng et al.) and real-world quadruped robot, which are illustrated in Figure 12
Table 19: Dynamics shift for Hopper, Walker2d, and Halfcheetah tasks. For the body mass shift, change the mass of the body in the source MDP M'. For the joint noise shift, we add a noise (randomly sampling in [-0.05, +0.05) to the actions when we collect the source offline data, i.e. D' {(s. s')) dp (s) my (als)r(s, (s''ss a noise).
Hopper
Walker2d
HalfCheetah
Body Mass Shfit
Joint Noise Shift action[- []nnoise
Body Mass Shfit mass(-1]=1.47
Joint Noise Shift actione-I]+noise
Body Mass Shöt mass[4]=0.5
Joint Noise Shift action[-I]+noise 1]+noise action[-1]44
Source Target
mass[-1]=2.5
muss[-1]=5.
action[-1]40
mass[-1]=22
action[-1]40
muss[4]=1
In the Hopper, Walker2d and HalfCheetah dynamics adaptation setting, set the D4RL (Fu al. 2020) dataset as our target domain. For the source dynamics, we change the body mass (body mass shift) add noises joints (joint noise shift) of the agents (see Table 19 for the details) and then collect the source offline dataset in the changed environment. Following Fu et al. (2020), on the changed source environment, we collect the 1) "Random" offline data, generated by unrolling a ran- domly initialized policy, 2) "Medium" offline data, generated by a trained policy with the "medium" level of performance in the source environment, 3) "Medium- Replllam""""""" (Medium-R) offline data, con- sisting of recording all samples in the replay buffer observed during training until the policy reaches the "medium" level performance, 4) "Medium-Expert" (Medium-D) offline data, mixing equal amounts of expert demonstrations and "medium" data in the source environment.
In the sim2real setting (for the quadruped robot), we use the Al dog from Unitree (Wang) 2020). We collect the target of- fline data using five target behavior policies in the real-world with changing terrains, shown in Figure 13 and collect the "Medium". ReedlumRRepaa" Expert" (Medium-E). "Mediun-Replay-Expp" E) source offline data in the simulator, where "Medium- Replay-Expert" denotes mixing equal amounts of "Medium- Replay" data and expert demonstrations in the simulator. In Section A.5] we provide the details of how to obtain the tar- get and source behavior policy, so as to collect our target and source offline data.
MMdiummR), "Medium-
(Medium-R-
Figure 13: Real-world terrains (for collecting the target offline data).
We list our tasks properties in Table 20 provide our collected dataset in supplementary material. implementation, we set = 0.1 for all simulated tasks and set 7 = 0.01 for the sim2real task. In Table 18 we also report the sensitivity of DARA on the hyper-parameters
A.S TRAINING THE (TARGET AND SOURCE) BEHAVIOR POLICY FOR THE QUADRUPED ROBOT
To obtain a behavior policy that can be deployed in simulator (for collecting the source offline data) or real-world (for collecting the target offline data), we introduce the prior knowledge (Iscen et al, 2018) and domain randomization (Tobin al. 2017 Peng et al. 2018).
26

---

Published as a conference paper at ICLR 2022
Table 20: Statisties for each task in our adaptation setting.
Environment
Dynamics Shift
Task Name
Target (IT)
Source (10S)
Random
10a (D4RL)
106
Medium
10 (D4RL)
106
Body Mass Shfit
Medium-Replay
20092 (D4RL)
10"
Medium-Expert
2 10 (D4RL)
2 106
Hopper
Random
10 (ID4RL))
106
Medium
10 (DARLL)
105
Joint Noise Shift
Redium-RRplay
20092 (D4RL)
106
Medium-Expert
10 (D4RL)
2 10
Random
10 (DARL)
10a
Medium
10 (D4RL)
106
Body Mass Shfit
Medium-
10093 (D4RL)
106
Medium Medium-Expert
2 10 (D4RL)
2 10
Walker2d
Random
10 (D4RL)
106
Medium
10 (D4RL)
106
Joint Noise Shift
Medium- Replay
10093 (D4RL)
10
Medium-Expert
2 10 (DARL.
2 10
Random
10 (D4RL)
106
Medium
10 (D4RL)
106
Body Mass Shfit
Medium-Replay
10100 (D4RL)
10a
Medium-Expert
2 10 (D4RL)
10
HalfCheetah
Random
10 (DARL)
105
Medium
10 (D4RL)
105
Joint Noise Shift
Medium-Replay
10100 (D4RL.)
105
Medium-Expert
2 * 10" (D4RL)
2 - 10c
Medium
3 10 (real-world)
10 (simulator)
Medium- Replay
3 + (real-world)
10 (simulator)
robot (Unitree)
Sim2Real
Medium-Ex Expert
3 10 (real-world)
2 x 10 (simulator)
Medium-Ree -eenn----ellllyaeeee
3 10 (real-world)
2 10 (simulator)
Prior Knowledge: To reduce the impact of the foot at the moment of touching the ground during the rohot locomotion, we designed a compound cycloid trajectory (Sakakibara et al., 1990) as prior knowledge. In our implementation for the foot trajectory, four aspects are mainly considered: 1) The robot walks stably without obvious shaking; 2) The joint impact of the robot during the locomotion small; The joint speed and acceleration of the robot during the locomotion are continuous and smooth: 4) The feet of the robot will not slide when they are in contact with the ground. Similar Lee al. (2020). we define a periodic phase variable ø; [0.0. 0.6). i 1,2,3, for each leg, which represents swing phase if o: E [0.0. 0.3) and contact phase if ø; E (.63, At every time step I, Φ = (L fo + dol + Ooffse (mod 2T... where Tm = 0.3, and fo 1.1 the base frequency, and Φo = [0, 0.3, 0.3, 0) is the initial phase. Domeet is part of the output of the controller. The trajectory of the swing leg is:
24 S
sin
+ So, 1,2 1,2
Tm 2
Tim
J; s
sin
2xt
S So. 3,4
Tm 2x
Tim
Yo
H
sgn
Tm
2
(2fe(()) - 1) + 1 + Zo
where
t
4 T
fE(t)
sin
T
4m
and
Ti
-1
0 1 < Tm
8gm
2
Tn < < <T 2
27

---

Published as a conference paper at ICLR 2022
The trajectory of the standing leg is:
27 Mpp! +
sin
2xt
+ Sp, 1,2
27 m
Tm
S
+
sin
2xd
S + So, i = 3,4
I'
y = Yo
Zo
where S = 0.14m, H 0.18m are the maximum foot length and height. So [0.17, 0.17, -0.2, 0. Yo = [-0.13, -0.13, 0.13), Zo = [-0.32, 0.32, -0.32, -0.32) are
the default target foot position in body frame.
Domain Randomization: To encourage the policy to robust to variations in the dynamics, we incorporate the domain randomization. In Table 21 provide the dynamics parameters and their respective range values.
Table 21: Dynamic parameters and their respective range of values utilized during training.
Parameter
Range
Mass
[0.95, 1.1] xdefault value
Inertia
[0.80, 1.2] xdefault value
Motor Strength
[0.80, 1.2] xdefault value
Latency
[0, 0.04] y
Lateral Friction
[0.5, 1.25] Ns/n
Joint Friction
[0, 0.05] Nm
The action is 16-dimensional vector consist- ing of leg phase and target foot position residuals in the body frame. The design of state space and reward function mainly follows the prior work Lee al. (2020). In Table 22 we provide the state representation.
State Space, Action Space and Reward Function:
Table 22: State representation for the behavior policy.
Data
Dimension
Desired direction
( (P
2
ay
Euler angle(rpy)
3
Base angular velocity 1 k?w)
3
Base linear velocity (FB")
3
Joint position/vclocity
08
24
FTG phases((sn) ((:) cos (o,))
8
FTG frequencies(
Base frequency fa)
1
Joint position error history
24
Joint velocity history
24
Foot target history (r1,a))-1,1-2)
24
The reward function defined as 0.Irie + 0.05r, + 0.05r,p 0.005r, 0.02r + 0.025r, + 10-5 10-
The individual terms are defined follows.
1) Linear velocity reward Ntv
exp(-33)
0.2))
Upr < 0.2
Fly =
Upr 0.2 1
28

---

Published as a conference paper at ICLR 2022
where Upr Uxy Gay is the base linear velocity projected onto the command direction.
2) Yaw angle reward ry.
Ty == exp(-(y - 8))
(17)
where y and ý the yaw and desired yaw angle.
3) Roll and pitch reward Trp:
Trp : e1p(-115 E(0 (6 10, arccos
< P2Zi (0,0, 1)
> 3) -/22)²²
(18)
Pzz
where e are the roll and pitch angle. Pas position in world frame. The advantage of designing the target pitch angle in this way is to ensure that the body of the robot parallel to the supporting surface the stand legs, thereby ensuring that the robot can smoothly over challenge terrain, such as upward stairs.
P4 or P.= P2 i E [1, 4] the foot
4) Base motion reward Tb
exp( 1 1.5 (Vay Upr 0zy))) + exp(-1.5 Σ (
(19)
where Way are the roll and pitch rates.
5) Body collision reward Tbe
The == -bodyy footls
(20)
where Tood and foot the contact numbers of robot's body parts and foot with the terrain, spectively.
6) Target smooth reward Ts:
Te llf., 2fd,e-1 + fa,t-2l1
(21)
where fi = =t, 1,t 2) are the target foot positions in the time-step t, f - and f - 2.
7) Torqure reward ry:
Fr =
Σ 1
(22)
where т; is the joint torques.
Training Details: Both the behavior policy and value networks are Multilayer Perceptron (MLP) with 3 hidden layers, which have 256, 128 and 64 nodes. The activation function the Tanh func- tion, and the optimizer is Adam. With the above prior knowledge, domain randomization and reward function, we train our behavior policy with SAC (Haarnoja et al. 2018) in PyBullet (Coumans & Bai 2016-2021).
A.6 ADDITIONAL RESULTS
Here provide additional results regarding the error bars (Tables 23 and 24
29

---

Published as a conference paper at ICLR 2022
1T+10S (DARA)
4.5 4.9
28,9 5.5
4.2 3.5
26.6 =
2.1 1.7
10.7 5.1
8.4 3.5
5.8 5.8 2.3
2.4 2.
17.2
1.5 0.3
53.3
-0.1 0.2
0.2
1.
1422
1
Normalized scores for the D4RL tasks (with body mass shift). We take the baseline results (for 10T) of MOPO from their original papers and that of the other model -fee methods (BEAR, BRAC-p, AWR, BCQ and CQL) from the D4RL paper (Fu et 20200
1T+10S
w/o Aug.
0.7 3.4 0.7
20.8 +
4.1 4.1 1.7
26.8 04
2
$ 5
5.5 4.6
4.8 4.8
17.1 0.2
1.6 0.4
52.5 1.2
0..1 0.1
5.7 5.7 4.7
3.1 2.4
5.5 3.5
AWR
MOPO
AWR
2
MOPO
10.4
10.3 10.3 0.3
+ 0.4
4.9
1.3
4.8 2.4
1.6
14
74
3
-022 -.2
5.7
5.3 #39
IT
4.1 1
1.6
7 3.6
30.9
8.8
27
1.3
10T
10.2
35.9
28.
27.1
11.7
28
67.5
23.7
1.5
17.4
15.5
53.8
13.6
17.8
44.6
59.3 12.2
99.7 16.4
1T+10S (DARA)
11 0.6
32.9 7.5
30.8 1 4.9
4 8.5
104
3.7
3.2 4.
78 3.1
18.6 6.5
77.5
3.4
81.7 8.77 3.1
93.3 8.8
2 2 1.5
34.7
21.6
64.1 10.8
63.5 22.5
1T+10S w/o Aug.
9.6 9.6 3.3
29.2 2 2.1
20.1 1 4.8
32.3 7.8
10.2 122-00 0.3
44.9 2.7
1.4 1.3
1.3 1.7
70 10.10.1
3.2 3.2 1.4
80 10
0.8 0.8 0.5
BRAC-p
BRAC-P
CQL
9.9
CQL
34.5 1
34.5
20.6 16.8
42.9 4
4
10.9 10.9
29 = 6.2
5.4 1.3 3.3
10.6 166
43 13 9.2
9.6 +
6.4 9.9
855 2.2
1.7 1.713
4.6 3.9
IT
0
10T
11
32.7
0.6
1.9
10.8
58
48.6
98.7
-02
77.5
-0.3
76.9
7
79.2
26.7
111
IT+10S (DARA)
8.4 1.2
0.5
17222
38.4 1.8
32.8 0.9
84.2 9.8
3.2 0.4
0.3 0.7
7.3 1.3
2.3 2.2
4.8 0.3
5.
15.1 0.2
57.2
1.6 1.611
9.7
1T+10S
w/o Aug.
4.6 1 3.4
0.9 0.3
18.2 1
8.3 0.3
25.7 5.5
28.7 + 1.9
75.4 7.8
3.1 + 0.9
0.6 0.5 0.5
6.5 5.1
1.5 2.5
4.5 0.5
50.9 4.3
14.9
55.2 + 3.8
0.6 0.6 =
BEAR
BCQ
BEAR
BCQ
IT
1 = 0.5
0.8 = 0
1.3 1.5
0.1
10.6 0.1
+ 6.3
9.3 4.4
162
1.5 1
-0.5 0.3
0.7 0.6
-0.1 0.1
0.9
32.8 8 8.2
0.6
32.5 9.1
9.1
0.8
37.1
58
1.8
6.9
10T
11.4
52.1
33.7
96.3
10.6
54.5
33.1
110.9
7.3
59.1
19.2
40.1
4.9
53.1
15
57.5
Body Mass Shift
Medium-R
Medium-E
Medium-R
Medium-E
Medium-R
Medium-E
Medium-R R
Medium-E
Random
Medium
Random
Medium
Random
Medium
Random
Medium
Table 23:
doddey
foppct
MalKkrza
30

---

Published as a conference paper ICLR 2022
26.3 18.4
1844
17.2 0.2
16.4 4,9
1T+10S (DARA)
4
41.3
7.2 0.2
0.2
27 0.1
2.9 2. 1.5
17.3
6.4 0
7.5
1.5 1.202
2.1 1
53.6 0
-0.2 -0.2
1222
Table 24: Normalized scores for the D4RL tasks (with joint noise shift). We take the baseline results (for 10T) of MOPO from their original papers and that of the other model foodeel-reeeeee free methods (BEAR. BRAC-p, AWR, BCQ and CQL) from the D4RL paper (Fu et 20200.
4 12.8
IT+10S w/o Aug.
AWR
3.6 2 0
38.8 88.8
6.7 1.2
27 7740
MOPO
1.3 1310. 0.5
9.2 + 5.4 5.4
2.3 4
1.4
AWR
1.5 1.6
0.6 17.2
1.4 0
53 5.1 0.1
MOPO
0.2
7.8 1 6.2
9.3
12.6
12.6
IT
10.1 10
0.1
12.4 1
25.5 + 1.2
1.5 +
2.7 2.7 221
0.8 0.8 0.1
1
0.9 0.1
12.2
40.4 4
-0.3 -0.3 0.1
5.8 5.9
0.8 # 0.7
2.9 3.1
30.3
12.2
6
10T
10.2
35.9
28.4
27.1
11.7
28
67.5
23.7
1.5
17.4
15.5
53.8
13.6
17.8
39
44.6
IT+10S (DARA)
10.8 2 0
101.4 1 0.2
87.8 13.3
10.8 0
3.6 0.6
108.9
0.7
72.9 9.1
34.8 + 10.5
1.8
0.6
81.2
1.8 0.4
116.5 9.1
37.6
58 +
8.8
IT+10S w/o Aug.
BRAC-p
10 +
27.6 2.8
89.9 7.8
23.4
104 0.4
58 +
2.6 26400 0.3
1..415
BRAC-P
3 2.9
55.2 15.88
32.1 + 11.9
62.3 + 13.1
2. 0.2 7
1.4 0.4
109.9 4.5
57.6
CQL
CQL
- 0.2
+ 4.8
1 6.4
+ 14
1 0.1
46.2 + 11.9
6.4
26.9
2.8 2.8
28.8 2
6.3 1.2
21.8 18.4
21.7
1.8 1.2
46.8 1 40
IT
0.5
10.8
26.6
13.4
19.8
1044
13.6
50.7
10T
11
32.7
0.6
1.9
10.8
58
48.6
98.7
-0.2
77.5
-0.3
76.9
7
79.2
26.7
111
IT+10S (DARA)
4.2 3.6
2 0.1
9.9 1 6.3
0.6
9.6 1
54.4 0.1
32 0.9
109 0.2
2.6 0.3
0.1 0.3
10.4 24
0.6 0.6
5.2 0.3
52.7 0.3
14.6 0.4
57.2 0.2
23.8 3.2
10.4
IT+10S
w/o Aug.
7.4 0.5
3.6 1.6100 0.4
0.8 +
49 1.7
96 0.5
0.6 + 0.1 06 0.6 0
4 0.2
0.8 0.4
3.404
44.9
9.8 522
7+0
BEAR
2
BCQ
BEAR
BCQ
18.6
IT
0
= 0
2.7 1 1.6
+ 0.2
+ 0.1
+: 2.2
13 4
44.6
2,2 1 0.1
-0.4 0.1
0.4 04
-0.2
3.7
43 1
8.3
5,7
4
3.6
10.5
45.8
0.6
0.8
0.8
10T
11.4
52.1
33.7
96.3
10.6
54.5
33.1
110.9
7.3
59.1
19.2
40.1
4.9
53.1
15
57.5
Joint Noise Shift
Medium-R
Medium-E
Medium-R
Medium-E
Medium-R
Medium-E
Medium-R R
Medium-E
Random
Medium
Random
Medium
Random
Medium
Random
Medium
uoddey
Eoddcy
Maaikcrzd
31


Reviews:
[{"review_id": "_FRDpp-qkT1", "reviewer": "Reviewer_73ni", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. The paper is clearly written and easy to understand.\n2. DARA appears to be the first method that addresses the problem of domain adaptation in the offline RL setting.\n3. The authors show extensive empirical results of DARA on both simulated and real-world experiments and evaluate the performance of DARA combined with most of the prior offline RL methods and observe reasonable improvement.\n\nCons:\n1. I think the algorithm is a bit incremental. It is in some sense a combination of Eysenbach et al. 2021 and conservative penalties introduced in prior model-based and model-free works. It seems to be a bit straightforward and therefore the novelty is a bit limited.\n2. DARA introduced additional hyperparameters compared to prior offline RL papers, i.e. the coefficient that controls dynamics shift, which could make the method hard to tune. DARA also requires training two classifiers, which also add another layer of complexity in terms of reproducibility. I also didn't see the mention of the hyperparameter settings. It would be important to know how the authors select hyperparameters for all of the environments since hyperparameter tuning in offline RL is usually expensive and potentially dangerous. It is also important to see the error bars of the results.\n3. The real-world experiments seem to use state inputs. It would be really intriguing if the authors could show good transfer results with image inputs in the sim-to-real setting.", "summary_of_the_paper": "This paper presents a new offline RL algorithm that aims to adapt the policy to a target domain with insufficient data with the help of abundant data in a source domain. The authors propose their method, DARA, which applies a reward penalty that correct the dynamics shift between the source and target domains. The authors combine DARA with previous model-based and model-free offline RL methods and show that DARA is able to achieve performance improvement over prior offline RL methods without the reward augmentation in both simulated and real-world tasks.", "summary_of_the_review": "I think the paper proposes a new algorithm that tackles a somewhat new problem, but the methodology lacks originality and the hyperparameter details are missing. Therefore, I would vote for a weak reject.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ZPOqFyoVJNL", "reviewer": "Reviewer_i5hn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nStrength:\n\n* Overall, the paper attempts to answer important question in offline reinforcement learning. \n* The given empirical evaluation seems theoretically grounded, results seem reasonably significant, with improved performance over the baseline source dataset.\n\nWeakness:\n* The motivation of the proposed approach needs to be expanded further. There is a single line which is hard to fully understand. \"Motivated by the (approximate) equivalence1 between the dynamics change and the rewards mod- ification (Kumar et al., 2020b; Eysenbach & Levine, 2019; Eysenbach et al., 2021), we propose to modify rewards in the source offline data to compensate for the dynamics shift. \"\n\n\n* Clarity of writing is a major weakness of the paper. It's not very well organized, logical flow is missing. As one example, the \"Related Work\"(Section 5) is towards the end of the paper, after the approach has been extensively described. A few more highlighted below to fix:\n\n* \"This modification aims to discourage the learning from these source offline transitions (next-state) that are hard to be achieved in the     target environment\": Can the latter part of the statement be more precise? This could be misinterpreted as a goat state to be.\n\n* Algorithm 1 Step 1: Perhaps clarify both the models are learnt independently for each of the datasets, ie 4 models.\n\n* Section 4.1: What does it mean for a trajectory to be \"optimal\"? In reality, is this actually binary?\n\n* Section 4.1 Figure 2 isn't clear to understand\n\n* Section 6.1 There is no dashed green line in Figure 3, as mentioned in Section 6.1\n\n* Section Was the source(noisy) dataset also collected from a D4RL policy?\n\n* \"Specifically, we out- line state-action-next-state: besides characterizing the state-action distribution shift, we additionally penalize the agent with a dynamics-aware reward modification.\": This line isn't very clear to me.\n\n* Section 4.1: Typo: \"the limit\"\n", "summary_of_the_paper": "The given work proposes the addition of a dynamics aware framework for Offline Reinforcement Learning. It motivates that  offline reinforcement learning suffer from a dynamics shift in SARSA transitions, which need to be effectively handled. They propose modifying reward in source offline dataset, penalizing it with the dynamics aware reward transition, setting the lower bound. The reward augmentation is done with explicit policy constraints as well as implicit policy constraints.They evaluate this approach on various robotic manipulation task from the Gym-MujoCo suite, evaluating and showcasing perforomance on four offline RL algorithms.\n\n", "summary_of_the_review": "Overall, the given work is theoretically grounded, providing incremental improvement over the past work in this area of tackling distribution shifts for offline RL learning. The results seem promising. However, the paper is not the easiest to read and interpret.  I'm willing to reconsider my decision, based on the feedback addressed by the authors.\n\nEdit: Updated score based on reviewer feedback\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Dcjk-f24pF", "reviewer": "Reviewer_wSxQ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Pros\n* The proposed framework is well justified for both model-free and model-based approaches. And the framework is easy to implement.\n* Figure 3 clearly shows that the reward penalty obtained through domain classifiers can aid the knowledge transfer when the dynamics shift.\n* The propose method has been tested on real environment and show improvement over the baseline aggregation strategy. This highlights the importance of dynamics-aware transfer in offline transfer RL for practical applications.\n\n## Cons/questions/suggestions\n* __Relationship to importance sampling__ The proposed reward compensation term is strongly related to importance weights. What if one simply weight each of the original offline data point based on the importance weights and train a policy using the combined importance weights source dataset and target dataset? Do you expect any advantage of the proposed method over importance weighting? \n* __Explanation for Lemma 1__ Could the authors elaborate on Lemma 1? Even if the environment has deterministic transition dynamics, given a finite randomly sampled dataset $\\mathcal{D}$, the learned dynamics $\\hat{T}$ model could still be different from the true dynamics $T$. Consider a simple tabular setting where there is a single initial state, a single allowed action, and two different states that could transition from this initial state and the only allowed action with equal probability. If I was given a dataset with ten state-action-next state tuples, and four of them correspond to the transition state 1 while the other six contain the transition state 2. Then I will get a wrong estimate of the environment dynamics. Please correct me if my understanding of the dynamic shift is incorrect. \n* __Impact of offline dataset quality__ Given different offline dataset quality, the proposed framework's improvement over the simple aggregation of source and target should vary. It would be nice for authors to compute the average improvement for all six algorithms over Hoper and Walker2d and see if there's any consistent trend. If so, it is important to discuss what insights we could learn from the results.\n\n## Minor comments\n* Could the authors comment on the sensitivity of the algorithm on the hyperparameter $\\eta$ in $r_{t} \\leftarrow r_{t}-\\eta \\Delta r$? How would one pick the value of $\\eta$ in practice? ", "summary_of_the_paper": "Offline reinforcement learning could enable one to train a data-driven decision making engine from purely offline data. However, the performance of polices learned through existing offline RL methods heavily depend on the quantity and quality of the offline data. When the data is scarce for a task, existing offline RL methods trained on limited offline data could perform poorly. \n\nThis work tackles the data efficiency issue of offline RL by transferring knowledge from a source offline dataset to a target offline dataset following an environment dynamics different from the source offline dataset. The proposed method augments the reward function by penalizing the state-action-next state transition tuples that are rare in the source environment. This reward augmentation strategy directly accounts for the dynamics shift between the source and the target domain. On several simulation a real-world tasks, the proposed method significantly improved the data-efficiency of existing offline RL algorithms. ", "summary_of_the_review": "This well-written paper studies the important problem of transferring knowledge to improve sample efficiency for offline RL. The proposed reward augmentation framework is well justified for both the model-free and model-based settings from the perspective of variational lower bound and model-bias when the dynamics shift, respectively. Also, the experiment results support authors' claims well. Overall, this is a good submission that can inspire future works along this direction.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "26t1Z29fA7r", "reviewer": "Reviewer_QyWv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper shows several strengths:\n\n(+) The method proposed in the paper contributes to improving the data efficiency in the offline reinforcement learning setting. It both reduces the amount of data needed from the target environment and relaxes the requirement on the reward information contained in the target environment’s dataset. \n\n(+) The theory behind the idea is well explained. The blue and red color letters help distinguish between the source and target MDPs.\n\n(+) The implementation of the method is also discussed, making the method applicable. The hyperparameter settings are reported in the appendix.\n\nIn the meanwhile, there exist concerns and limitations:\n\n(-) My concern comes from the experiment section. The experiment results are averaged over 3 seeds only, which does not give an accurate estimation of the true performance. Even though there are 4 environments with 4 dynamic shift cases each and DARA improves the performance in most cases, the risk still exists that the averaged performance of one single test is inaccurate. Therefore it could be better if more random seeds can be tested.\n\n(-) It seems like the proposed method is limited to the case that the target environment is restricting transitions that exist in the source task. It is not clear to me if the method still helps when the target MDP adds new transitions. For example, in the Map task, collecting source dataset with the obstacle and collecting target dataset without the obstacle. In this case, it can be harder for the framework to capture the change in the transition dynamics, thus harder for the agent to figure out the new optimal policy (the shorter path without the obstacle). If this is true, the cases that DARA can be applied to might be limited, though I believe it is still helpful in cases like proposed in the paper.\n", "summary_of_the_paper": "The paper focuses on the dynamics shift problem in the offline reinforcement learning setting, with the assumption that the estimated MDP from the offline dataset is deterministic. It proposes DARA, a framework that relaxes the requirement of data (both the amount of data and the reward information contained in the data) needed from the target task. By capturing the dynamics shift between the source and target environments, DARA modifies the reward gained in the source environment to discourage transitions that have a smaller probability to happen in the target environment. The framework works with both model-free and model-based settings. ", "summary_of_the_review": "I recommend acceptance. The paper introduces a framework to adapt to the target environment with a relaxed requirement on data from the target task. I believe this work contributes to improving the data efficiency in the offline reinforcement learning setting, even when the access of the reward function in the target environment is limited. There exists shortages and limitations as listed, but overall, I consider the positives to outweigh the negatives.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: **Summary**

This paper proposes a method to do a sample-efficient offline domain adaptation method where the setting requires one to have abundant amount of data from the source domain but limited amount of data available in the target domain. The proposed approach DARA achieves that by accounting for the the dynamics shift between the source and the target domain via a reward penalty. The paper shows promising experimental results.

**Final Thoughts**
Overall the paper is well-written, and it addresses an important problem. The reviewers are mostly positive about the paper at the end. The authors did a very good job addressing the concerns raised by the reviewers. Reviewer 73ni had concerns about the novelty of the approach, it would be nice if the paper can make it more clear about the novelty of the proposed approach compared to the other existing methods in the paper. I  would also recommend the authors to incorporate the feedback and the suggestions made by the reviewer into the camera-ready version of the paper.
Decision: Accept (Poster)