When to Trust Your Simulator: Dynamics-Aware Hybrid Offline-and-Online Reinforcement Learning
Authors: Haoyi Niu
Shubham Sharma
Yiwen Qiu
Ming Li
Guyue Zhou
Jianming HU
Xianyuan Zhan
Venue: NeurIPS 2022 Accept (2022)

Abstract: Learning effective reinforcement learning (RL) policies to solve real-world complex tasks can be quite challenging without a high-fidelity simulation environment. In most cases, we are only given imperfect simulators with simplified dynamics, which inevitably lead to severe sim-to-real gaps in RL policy learning. The recently emerged field of offline RL provides another possibility to learn policies directly from pre-collected historical data. However, to achieve reasonable performance, existing offline RL algorithms need impractically large offline data with sufficient state-action space coverage for training. This brings up a new question: is it possible to combine learning from limited real data in offline RL and unrestricted exploration through imperfect simulators in online RL to address the drawbacks of both approaches? In this study, we propose the Dynamics-Aware Hybrid Offline-and-Online Reinforcement Learning (H2O) framework to provide an affirmative answer to this question. H2O introduces a dynamics-aware policy evaluation scheme, which adaptively penalizes the Q function learning on simulated state-action pairs with large dynamics gaps, while also simultaneously allowing learning from a fixed real-world dataset. Through extensive simulation and real-world tasks, as well as theoretical analysis, we demonstrate the superior performance of H2O against other cross-domain online and offline RL algorithms. H2O provides a brand new hybrid offline-and-online RL paradigm, which can potentially shed light on future RL algorithm design for solving practical real-world tasks.

When to Trust Your Simulator: Dynamics-Aware Hybrid Offine-and-Online Reinforcement Learning
Haoyi Niu Shubham Sharma" 1.2 Yiwen Qiu Ming Li
Guyue Zhou', Jianming Hu! Xianyuan Zhan 1,5t
Tsinghua University, Beijing. China
Indian Institute of Technology. Bombay, India
Shanghai Jiaotong University, Shanghai, China
Beijing National Research Center for Information Science and Technology, Beijing, China
5 Shanghai AI Laboratory, Shanghai, China
(t da. thu, shubh. aml 1072, inwg
@mmall com
hu
cn
{ (zhouguyue, zwa wtw.aaaannnnat...n.in.ieeeio.aee edu. en
Abstract
Learning effective reinforcement learning (RL) policies to solve real-world complex tasks can be quite challenging without a high-fidelity simulation environment. most cases, we are only given imperfect simulators with simplified dynamics, which inevitably lead to severe sim-to-real gaps RL policy learning. The recently emerged field of offline RL provides another possibility to learn policies directly from prelccelected historical data, However, to achieve reasonable performance, existing offline RL algorithms need impractically large offline data with sufficient state-action space coverage for training. This brings up new question: it possible to combine learning from limited real data in offline RL and unrestricted exploration through imperfect simulators in online RL to address the drawbacks both approaches? In this study, we propose the Amamicc-AAaat Hybrid Offline- and-Online Reinforcement Learning (H20) framework to provide an affirmative answer to this question. H2O introduces dynamics-aware policy evaluation scheme, which adaptively penalizes the Q-function learning on simulated state- action pairs with large dynamics gaps, while also simultaneously allowing learning from a fixed real-world dataset. Through extensive simulation and real-world tasks, well theoretical analysis, we demonstrate the superior performance of H2O against other cross-domain online and offline RL algorithms. H2O provides a brand hybrid ofdineeand-online RL paradigm, which can potentially shed light on future RL algorithm design for solving practical real-world tasks.
1 Introduction
Over recent years, criticism against reinforcement learning (RL) continues to pour in regarding its poor real-world applicability. Although RL has demonstrated superhuman performance solving complex tasks such as playing games [Mnih et al., 2013; Silver et al., 201 its success is heavily dependent on availability of an unbiased interactive environment, either the real system or a high-fidelity simulator, as well as millions of unrestricted trials and errors. However, constructing high-fidelity simulators can be extremely expensive or even impossible due to complex system dynamics and unobservable information in the physical world. Less accurate simulators with simplifications, on the other hand, are easy to build, however, also lead to severe visual and dynamics sim-to-real gaps when learning
"Work done during internships at Institute for Al Industry Research (AIR), Tsinghua University. 1 'Corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Offline Dataset
Real-World System
Imperfect Simulator
Offline RL
Hybrid Fifline-add-Oniigg RL
Online RL
x A
xxx A
Dataset
Datoset
Sim data
Limited Coverage
Broad Coverage
Corrected Dynamics
Dynamics Gap
Figure 1: Conceptual illustration of the dynamics-aware hybrid offline-and-online RI. framework
RL policies [Rao et al., 2020; Peng et al., 2018]. Among different sources of sim-to-real gaps, visual gaps are relatively well-addressed in number of methods, e.g. through domain randomization [Tobin et al., 2017], proper model setups that are less impacted by visual gaps [Lee al., 2021], using a robust and transferable state-encoder [Bewley et al., 2019; Rao el al., 2020; Wang el al., 2022]. Dynamies gaps the other hand cause a systematic bias that not easy to address through modeling process. These dynamics gaps widely exist in almost every imperfect physical simulator with simplified dynamics, posing great challenges for RL when solving
tasks.
Recently, offline RL provides another possibility to directly learn policies from rorl-world data [Levine et al., 2020; Fujimoto et al., 2018; Kumar et al., 2019]. which has already been used in solving a number of practical problems, such as industrial control [Zhan et al., 2022a], robotics (Lee er al., 2021] and interactive recommendation [Xiao and Wang. 2021]. However, the policies learned through existing offline RL algorithms are often over-conservative due to the use of data-related policy constraints [Fujimoto el al., 2019; Kumar al., 2019; Fujimoto and Gu, 2021], value regulariza- tions [Kumar et al., 2020; Kostrikov el al., 2021; Xu el al., 2022b], or in-sample learning [Kostrikov al., 2022: Xu al., 2022a] combat the distributional shift [Kumar al., 2019]. This hurts the performance of offline RL policies and makes their performances strongly depend on the size and state-action space coverage of the offline dataset. In real-world scenarios. data collection can be expensive or restrictive. The state-action space coverage of actual offline datasets can be quite narrow, which directly limits the potential performance of offline RL policies.
Despite all the drawbacks, online RL with an imperfect simulator and offline RL with the real- world dataset bear indispensable potential when jointly used to learn pergrperffrmecce policies and overcome the sim-to-real issues. Within a simulator, online RL agents can perform unrestricted exploration and have an access great quantity and diversity of state-action data, while offiine datasets may have poor coverage. On the other hand, real-world datasets contain perfect dynamics information, which can be readily used to guide and correct erroneous information in online training with simulation data. This intuition has motivated researchers to explore how to incorporate offline real-world data with simulation-basee to pure online [Eysenbach et al., 2020] or offline [Liu et al., 2022] learning settings, None of the prior works has successfully combined offline and online learning organically, while simultaneously handling the dynamics gaps between these two regimes.
online training, however, existing studies are only restricted
In this paper, present the first Hybrid Offline-and and-Online Reinforeement Learning framework (H20) as conceptually illustrated in Figure 1, which enables simultaneous policy learning with offline real-world datasets and simulation rollouts. The core of H2O is the introduction of dynamics-aware policy evaluation, which adaptively pushes down or pulls up Q-values on simulated data according to the dynamics gap evaluated against real data. We provide analyses and insights on why such dynamics-aware learning can potentially bridge the sim-to-real gap. Finally, we analyze the impact of each design component in H20 and demonstrate its superior performance against other cross-domain RL algorithms through extensive simulation and real-world experimental validations.
2 Related Work
2.1 Reinforcement Learning Using Simulators with Dynamics Gap
Dealing with dynamics gaps in simulators has long been recognized as a challenging task. The most straightforward approach is to calibrate the simulated dynamics with the real-world counterpart using

---

system identification methods [Ljung, 1998; Rajeswaran et al., 2016; Chebotar et al., 2019]. These methods typically need plenty of offline or even real-world interaction data tune the parameters of the simulator (Yu et al., 2017], which sometimes can be costly or offer little improvement if the simulator is over-simplified. Domain Randomization (DR) [Peng el al., 2018, 8; Andrychowiez. er al., 2020] provides another perspective that trains RL policies in a series of randomized simulated dynamics. DR has been shown to yield more adaptable policies in real-world, yet it often needs manually-specified randomized parameters and nuanced randomization distributions [Vuong et al., 2019]. More recently, Dynamics Adaptation, e.g., DARC [Eysenbach et al., 2020], augments the simulation reward r(s, a) with dynamics gap-related penalization term Ar(s' s, a) in online simulation- based RL training, derived from minimizing the divergence of distributions over real and simulation trajectories. DARC uses real data and trains two discriminators to evaluate Ar(s' s, a). rather than directly leveraging the data for RL training. In a similar vein, DARA [Liu et al., 2022] is the pure offline version of DARC, sacrificing the unlimited online exploration in the simulated domain. While existing RL studies have tackled the dynamics gap issue from different perspectives, none of them is able develop a structured framework by combining offline data with online learning, simultancously addressing the dynamics gap.
2.2 Offline Reinforcement Learning
The recently emerged offline RL methods provide a new alternative to learning policies directly from a fixed. pre-collected dataset. A key challenge of offline RL is the distributional shift problem [Kumar el al., 2019; Levine et al., 2020], which is caused by counterfactual queries of function approximators (e.g., value function, police network) on -fittri-iisiitttioo (OOD) samples. Conventional off-policy RL methods will suffer severe value overestimation due to distributional shift. Existing offline RL methods solve this problem by introducing various data-related regularizations to constrain policy learning from deviating too much from the offline dataset, such using data distribution, support or dataset distance based policy constraints [Fujimoto et al., 2019; Kumar et al., 2019; Wu et al., 2019; Fujimoto and Gu, 2021; Xu et al., 2021; Li et al., 2022a], regularizing the value function on OOD data [Kumar et al., 2020; Kostrikov et al., 2021; Yu et al., 2021; Li et al., 2022b; Xu et al., 2022b], samsamlle learning [Kostrikov et al., 2022; Xu et al., 2022a], and adding uncertainty- beaseaddtt-bbeedd penalties (Yu et al., 2020; Kidambi et al., 2020; Zhan el al., 2022a.b]. Conservatism and pessimism central principles [Buckman et al., 2020] used in most existing offline RI. algorithms, which cause the performances of these methods to heavily depend on the size, quality, and state-action space coverage of the offline dataset. In most practical settings, getting dataset with large coverage is impractical. Meanwhile, if online interaction with a simulator is involved, although may not be accurate, it can substantially supplement the state-action space coverage of offline datasets in policy learning. This motivates a new paradigm of combining both offline and online RL, which is explored in our proposed H2O framework.
Background
3.1 Reinforcement Learning
We consider the standard Markov Decision Process (MDP) setting, which is specified by tuple M := (S, A. r, PM- 7), where S and A are the state and action spaces, r(s, a) is the reward function, PM (s' |s, a) refers to the transitional dynamics under M. p(s) depicts the initial state distribution, and 7 E (0, 1) is the discount factor. The goal of RL is to learn a policy I (als)) that maps state to action so to maximize the expected cumulative discounted reward starting from 0, i.e., J(M, π) = Fe Eparera (: (e) PM.-ls. (s.a) IE 7rr (St, a:)] In conventional actor-critic formalism [Barto al., 1983; Sutton el al., 1998], one learns an approximated Q-function Q by minimizing the squared Bellman error (referred as policy evaluation), and optimizes the policy by maximizing the Q-function (referred as policy improvement) as follows:
(Q(s.a) 10---1 BT (Q(s,a)-B**(0) ((s.ss))) 21
Q f- arg min 22
(Policy Evaluation)
(1)
Q
A arg max E [((s.a)) a)
(Policy Improvement)
(2)
where U can either be the replay buffer B generated by previous version of policy t through online environment interactions, or can be fixed dataset D (ss) af s }." as in offline
3

---

RL setting. 8d is the Bellman operator, which is often used as the Bellman evaluation operator Td ((s a) = r(s, a) + (Barra(a)ss))) [ (s'.aa
3.2 Offline RL via Value Regularization
In the offline setting, performing the standard Bellman update Eq. (1) can result serious overestimation over Q-values due to distributional shift. An effective approach to alleviating this problem is to regularize the value function on OOD actions, which is introduced in CQL [Kumar er al., 2020]. CQL learns policy Ωt upon lower- bounded Q-function that additionally pushes down the Q-values on actions induced by ua(as) and pulls up Q-values on trustworthy offline data:
min max (Eg-D.ar-( IQ(s, a)] E (Q(s, Eg a))) + & (Q,
(3)
where the dataset is collected by some behavioral policy Tp and ((-lss is some sampling dis- tribution, which is often taken as # (- in prior works [Yu et al., 2021; Li al., 2022b]. ( (Q,
denotes the Bellman error in Eq. (I). It is worth noting that the above value regularization framework has the potential to handle data from two sources with some preferences, i.e., unreliable pushed- down data and reliable pulled-up data. This motivates us to devise the H2O framework to perform simultaneous offline and online policy learning using similar value regularization recipe.
4 Hybrid Offline-and-Onling Reinforcement Learning
Naïvely combining offline and online data from different distributions may cause the mismatch between data distributions, giving rise issues similar to the distributional shift problem as in offline RL settings [Kumar et al., 2019]. Caution needs to be taken when dealing with trustworthy offline data from the real world and the potentially problematic simulated online rollouts (described in Section 4.1). Secondly, the dynamics model underpinning the simulator is only roughly aligned with the real-world counterpart, and the sim-to-real dynamies gaps can be heterogeneous across different state-action pairs. Hence uniform value regularization upon all simulated samples may not be appropriate fully leverage the potential of an imperfect simulator. well-founded dynamics gap measurement for adaptive value regularization is needed (Section 4.2). Lastly, it is also worth noting that the biased simulated dynamics could induce erroneous next-state predictions, which may lead to problematic Bellman updates. Such errors also need to be properly considered in the algorithm design (Section 4.3). We approach all the aforementioned issues with what we call Dynamics-Aww Hybrid Offline- and--nine (H20) RI., which is described in the following content.
4.1 Incorporating Offline Data in Online Learning
To address the potential mismatch of training data distributions from different data sources, existing online RL frameworks are not suitable, however, offline RL approaches such as value regularization can provide a viable foundation. We modify the previous value regularization scheme in offline RL and propose the dynamics- awaare policy evaluation, which has the following general form:
min max 33 [Eg.add (Q a)) E. (es.s a)] + R(d*)] E (Q. Q de
where 3 is a positive scaling parameter. ds a) is a particular state-action sampling distribution that associates with high dynamics-gap samples, and R(d)) is regularization term for dd enforce this designed behavior. Ideally, we would want to penalize Q-values at the simulated samples with high dynamics gaps, rather than all of the samples. ε(Q, (5*() denotes the modified Bellman error of the mixed data from offline dataset D and the simulation rollout samples in online replay buffer B, which are generated by the real MDP M and the simulated MDP M respectively. Specifically, the Bellman error of the simulated data needs to be fixed. The terms marked in red and blue are the key design elements that account for the dynamics gaps in onfline-and-onliee RL policy learning, which will be discussed in the following sub-sections.
4.2 Adaptive Value Regularization on High Dynamics-Gap Samples
To achieve adaptive value regularization, can the minimization term Eg [Q(s, a)] to penalize the high-dynamics gap simulation samples, and use the maximization term Eg ((ss a)] to cancel the penalization on real offline data. The key question is how to properly design d a) to

---

assign high probabilities to high dighh-ynamicss gap samples. In our study, we leverage the regularization term R(dx) to control the behavior of (s a). We choose R(d²) 1 (s. a) (sv a)). where (x1(- is the Kullback-Leeible (KI.) divergence and (s. a) is distribution that char- acterizes the dynamics gaps for samples in the state-action space, Hence maximizing over R(d%)
draws closer between d and w, and achieves our desired behavior. Note that the inner maximization problem over d in Eq. (4) under this design corresponds to the following optimization problem:
max E (x.n) (Q(s. a)] Dx (d (s).aals a)) s.t. Σ da
(s. a) 1, d (s. 0 (5)
Above optimization problem admits closed-fone solution d (s. a) w(s, a) exp (Q(s. (see the derivation in Appendix A. Plugging this back into Eq. (4), the first term now corresponds to a weighted soft-maximum of Q-values at any state-action pair and the original problem transforms into the following form:
min 3 log Σ w(s a) exp (Q)s - Eg [Q(s. + + (Q.
(6)
8,8
This result is intuitively reasonable, as we are penalizing more on Q-values with larger w(s, a). corresponding those high dynamics- gap simulation samples. Then the next question is, how do practically evaluate wss a) given the offline dataset and the online simulation rollouts? Specifically, we can measure the dynamics gap between real and simulated dynamics as u(s, a) == 1 ( (s') PM a)) = Ex'=Px: log(P, a)) a a)). w(s, a) can thus be represented as a normalized distribution of u, i.e., w(s, a) = u(s, a)/ Σι,a u(3, ā). Now, the challenge is how to evaluate the dynamics ratio PM
state-action pair (s,a)
PM. Note that according to Bayes' rule:
PM a)
p (s' (s, a, sim)
(s'ls real) = p a, s') (realls, a, a.s)
PM a,)
p [sim (s. a)
(real ss, a)
p (sim s, a, s')
p (sim s. a)
1 1-p(real) (real (s, a, s') 1 p (realls, a) p (real|s, a, a')
(7)
(real|s, a, a') (real|s,
(real]s. (s,
Hence, we can approximate p (real]s, s') and p (real]s, a) with a pair of discriminators (-ls, a, s') and De... (-|s, a) respectively that are optimized with standard cross-entropy loss between real offline data and the simulated samples as in DARC [Eysenbach et al., 2020].
4.3 Fixing Bellman Error due Dynamics Gap
Due to the existence of dynamics gaps in the simulator, directly computing the Bellman error for simulated samples in the online replay buffer B (i.e., En..,s'n1( - B () a))²) is problematic. This is because the next state s' comes from the potentially biased simulated dynamics P Pv., which can result in the miscalculation of target Q-values in Bellman updates. Ideally, we wish the next state s' comes from the real dynamics PM. however, such an s' given an arbitrary state-action pair (s, a) is not obtainable during training. To fix this issue, we reuse the previously evaluated dynamics ratio in Eq. (7) as an importance sampling weight, and introduce the following modified Bellman error formulation on both offline dataset D and simulated data B:
11-8))) E (Q..BOQ B BoaD [(e- [ Q))) [s.a)]² (s,)a)))
+ Bilg
[[0 8 Q)
(s, a))
2 (8)
1 BaawnD ((- (s, a))² + 1
} Pamann
PM a) Px (s [s,
(Q - Q) (s. a)
4.4 Practical Implementation
Combining Eq. (6) and (8), we obtain the final dynamics-awar policy evaluation procedure for H2O, which enables adaptive value regularization on high dynamics-gaj samples as well as more reliable Bellman updates. H2O can be instantiated upon common actor-critic algorithms (e.g., Soft Actor-Critic (SAC) [Haarnoja el al., 2018]). The pseudocode of H2O built upon SAC is presented in Algorithm 1, which uses the policy improvement objective from SAC and À is a temperature parameter auto-tuned during training. Other policy improvement objectives, or simply maximizing the expected valuues as in Eq. (2) are also compatible with the H2O framework.
5

---

Algorithm 1: Dynamics-Awart Hybrid Offline-and-Onling Reinforcement Learning (H20)
Data: an offline dataset D from the real world, an imperfect simulator with biased dynamics M Initialize: critic network Qo, target network Qg. actor network Tœ. replay buffer B = Ø for simulated transitions, discriminators De... (-|s, a, and Dæ,. a)
2 for step t = 1, .T'd do
B - B U ROLLOUT(T M);
> Collect simulated data
4
Pass t T Deas - VV oan CROSSENTROPYLOSSSBBB D, D, ssas):
Update Di Update Den,.
D sa t D V/V am CROSSENTROPYLOSS//
ssa):
0 0- 1Q V (log Σ ((s, a) exp (Qa(s, a)) Eg (Qu(s, + & ( BT
Φ ++ + Th v (Dun) [Qa (s. a) Alog {T
i % target update period 0 then
0- + (1 7)0 T0:
> Soft update periodically
10
end
end
For practical considerations, we introduce two relaxations in our implementation. First, in Eq. (6), we need to sample from the whole state-action space to compute the weighted average of exponentiated Q-values, which can be impractical and unnecessary since not all state-action pairs are of interest for policy learning. Instead, we approximate this value using the state-action samples in the mini- batch of the simulated replay buffer B. Second, the evaluation of the KL divergence in u(s, a) a) PM'lss a)) involves sampling next states s' from P P (-|s, a). which can
Eyt-rg log( (o be infeasible given black-box simulator. We take a simplified approach that approximates the expected value by averaging over N random samples from a Gaussian distribution ((ss, Σp), where ED is the covariance matrix of states computed from the real offline dataset. Although this treatment simple, we find it highly effective and produces good performance in empirical experiments. Please turn to Appendix B.1 for more implementation details.
Interpretation of Dynamics-Aware Policy Evaluation
In this section, we analyze H2O and provide intuition on how works when combining both offline and online learning. Interestingly, we find that the final form of (lynamics-awan policy evaluation as given in Eq. (6) and (8) actually leads to an equivalent adaptive reward adjustment term on the Q- functio,,, which depends on the dynamics gap distribution w(s, a) and the state- action distribution of the dataset M (s, a). To show this, note that the weighted log-sum-exp term log >... wss, a) exp (Q(s, a)) log E. (n,,a)((s.a) exp(Q(s, a)) in Eq. (6) can be bounded as follows (see Appendix A.2 for detailed proof):
Varw Texp(exp(Qrr a))]
E. (.. (s),aa) (Q(s < log (,,an exp(Q(s, a)) S an (Q(s, a)] + 2 eexp (2Q
(9)
where we denote the range of learned Q-values as [Qmin Qmaz), and Var (exp (Q(s, a)]] denotes the variance of exp(Q(s,, a)) under samples drawn from distribution w(s, a). The LHS inequality is a result of Jensen's inequality, and the RHS inequality is a direct result from [Liao and Berg, 2018]. Above inequalities suggest that Eg (gga-~(s.a)) [Q(s. a)] can be reasonable approximation of our weighted som-ssum-xpppppp term if the Q-function takes large values and the gap between exp((máán) and / (Var_/expl Q (s. a is not too large. This can be practically satisfied if we let o - and design the range of reward function as well as the episode done conditions properly.
With this approximation, we can consider the following approximated policy evaluation objective based on Eq (6) and (8), which offers a much cleaner form for analysis:
min 3 (B [Q(s, a)] - EE..DD [Q(s, a)) + (2.
(10)
The ablation study in Appendix C.I manifests that this approximated policy evaluation objective only yields mild performance drops against the original version most cases, making it reasonable substitution for theoretical analysis. Assuming that the Q-function is tabular, we can find the Q-value

---

corresponding to the new objective by following an approximate dynamic programming approach and differentiating Eq. (10) with respect to Q in iteration &o (see Appendix A.2 for details):
w(s a) - (s.a) d (s. - (s,
(u+ = (B (s).a) 3
(11)
where dM (s, and (s,(s, a) are state-action marginal distributions under behavioral policy Tp we represent the second term in the RHS of above equation
and the learned policy respectively. as v(ss, a), we can see that v(s, a) can be perceived as an adaptive reward adjustment term, which penalizes or boosts the reward at a state-action pair (s, a) according to the relative difference between dynamics gap distribution (s,, a) and marginal state-action distribution of the offline dataset d' (s. a). Specifically, we can make the following interesting observations:
If w(s,a) a) > d. (s. a), then (ss, a) serves as reward penalty. This corresponds to the case that either the state-action pair (s. a) has large dynamics gap, or it belongs to OOD or relatively low data density areas (dM (s, a) = 0 or < w(s a)). Under such cases, we push down the Q-values and adopt conservatism similar to the typical treatment offline RI. settings [Kumar el al., 2020]. The higher the dynamics gap is (larger w), the more we penalize the Q- p-function.
If ((s, a) d' (s, a). then we are evaluating at some good data areas with low dynamics gap or have more information from offline data to control the potential dynamics gap in the simulated online samples. Under such cases, v(s, a) serves a reward boost term and adopts optimism. Although it overestimates the Q-values, we argue that this can be beneficial. As it can promote exploration in these "good" data regions and potentially help reduce variance during training.
With the adaptive reward adjustment term v defined in Eq. (11), we can further show that the value function V(s) is underestimated high dynamics gap areas, which allows for safe policy learning in our
setting. Detailed theoretical analysis can be found in Appendix A.3.
6 Experiments
In this section, we present empirical evaluations of H2O. We start by describing our experimental environment setups and the cross-domain RL baselines for comparison. We then evaluate H2O against the baseline methods in simulation environments and on real wheel-legged robot. Ablation studies and empirical analyses of H20 are also reported. Additional experiment settings, ablations, and results can be found in Appendix B, C, and D respectively.
6.1 Experimental Environment Setups
Simulation- experiments. We conduct iimulation-based experiments in the MuloCo physics simulator [TToorov el al., 2012]. In particular, construct three new simulation task environments (serve as the simulated environments) with intentionally introduced dynamics gaps upon the original MaloCo-HallCheettal task environments (serve the real environments) by modifying the dynamics parameters: (1) Gravity: applying 2 times the gravitational acceleration in the simulation dynamics; (2) Friction: using 0.3 times the friction coefficient to make the agent harder maintain balance; (3) Joint Noise: adding a random noise sampled from A(0, 1) on every dimension of the action space, mimicking a system with large control noise. As for the offline dataset from the real world (original simulation environment), we use the datasets of the corresponding task from standard offline RL benchmark D4RL [Fu el al., 2020]. Specifically, we consider the Medium, Medium Replay and Medium Expert datasets, as in typical real-world scenarios, we do not use a random policy or do not have an expert policy for system control. The online training is performed on the modified simulation environment, and we evaluate the performance of the learned policy in the original unchanged MuJoCo environment in terms of the average return.
Real-world experiments. We also evaluate the performance of H2O on real wheel-legged robot that moves on a pair of wheels to keep it balanced, as illustrated in Figure 2(a). The state space of the robot is a quadratic-tuple S = (0, 0, xr, .i),, where 0 denotes the forward tilt angle of the body, is the displacement of the robot, ê and èt are the angular and linear velocity respectively. The control action the torque T of the motors at the two wheels. Here we construct two tasks for real-world validation: (1) Standing still: We want to make the robot stand still and not move or fall down; (2) Moving straight: We want to control the robot to move forward at target velocity e while keeping its balance. We record 100,000 human-controlled transitions with different reward functions for
7

---

Table Average returns for MuloCo-HalfCheetall tasks. Results are averaged over random seeds.
Dataset
Unreal dynamics
SAC(sim)
CQL((eal)
DARC
DARC+
H20
Gravity
4513=513
6066173
5011::456
5706+-400
7085::416
Medium
Friction
2684::266
6066173
6113=1104
6047=112
6848
Joint Noise
4137+805
6066
5484+111
5314++520
231
Gravity
4513=513
5774
5105:1460
4958:1540
6813:1289
Medium Replay
Friction
2684:12646
5774::214
5503:1263
12881100
5928:1896
Joint Noise
4137-805
57744:214
5137:1225
5230:1209
6747-1427
Gravity
4513=1513
3748+892
4759++353
12+
4707+7799
Medium Expert
Friction
2684+2646
3748+892
1088++1880
7989+93999
6745+5562
Joint Noise
4137-805
3748
5288.1104 104
7331767
5280.11329
both tasks, which serve as the offline dataset used in RL training. Additional real-world experiment settings like reward functions and dataset analyses are elaborated in Appendix B.2.
The control task is relatively challenging, the robot has to use two wheels to keep balance and can easily fall to the ground. The simulation environment is constructed based on Isaac Gym [Makoviy- chuk et al., 2021], depicted in Figure 2(b). The simulation environment suffers from intricate dynamics gaps. Notably, the electric motors in the real robot bear dead zones, which can lead to observational errors, whereas the simplified dynamics in simulation fails to capture this complicated situation. Besides, the rolling friction applied on wheels is well-simulated, while the sliding friction cannot be modeled realistically in Isaac Gym. Furthermore, the friction coefficient between wheels and the ground as well as its wear-and-tear, are not possible to be configured exactly as in the real-world environment. All these dynamics gaps can lead serious sim-to-real transfer issues when deploying RL policy learned in simulation to real-worke scenarios.
6.2 Baselines
As H2O the first RL framework under the hybrid offline-and- -nn-onine setting, we can only compare with some purely online or offline RL algorithms, as well as representative methods that partially incorporate offline data online or offline policy learning in a less integrated manner, which are:
SAC (sim) [Haarnoja al., 201 2018]: the SOTA off-policy online RI algorithm, which is trained within the imperfect simulator and evaluated in the original simulation or real-world environments.
CQL (real) [Kumar el al., 2020]: the representative offline RL algorithm using value regularization. We run CQL on D4RL and recorded real-world robotic control dataset, which is not impacted by the dynamics gaps, but suffers from limited state-action space coverage in the offline datasets.
DARC [Eysenbach al., 2020]: DARC uses reward correction term derived from a pair binary discriminators to optimize the policy within an imperfect simulator. We train these discriminators with real and simulation samples and learn policy within the simulation.
DARC+: variant of DARC. in which the real offline data are not only utilized for discriminator training, but also used in policy evaluation and improvement. DARC+ showcases the model behavior when naïvely combine offline and online policy learning under the DARC framework.
6.3 Comparative Evaluation of H2O in Simulation and Real- World Experiments
In Table 1, we present the comparative results of H2O and other baselines on the simulation- based experiments. Notably, H2O achieves the best performance in almost all tasks compared with the baselines. Due to the relatively large dynamics gap, directly training SAC in simulation yields the poorest performance in most of the tasks. Offline RL method CQL achieves reasonable performance as compared with the online RL counterpart, which achieves second-highess scores in Medium and Medium Replay tasks. DARC performs surprisingly very well in the Medium Sapert-Friction task, perhaps due to specific dataset and dynamics gap properties. DARC+ shows inferior performance compared with DARC in most tasks, indicating the insufficiency of naïvely combining offline and online policy learning. Nevertheless, H20 consistently provides superior performance under different offline datasets and dynamics gap settings, demonstrating the effectiveness of our proposed dynamics-awerr -ware policy evaluation scheme.
8

---

(a) Robot in real world
(c) Experimental results for standing still task
Displacement
Angle
044
Average Reward
(mm)
(radd
10
1220
SAC
-10
Sim
Real
time (5)
time (s)
(d) Experimental results for moving straight task
(b) Robot in simulation
Velacity
Angle
1.90
Average Reward
0.99
(rad))
0.3-
0.3-
-139
%
time (s)
time
R20
001
DARC
SAC
DARC-
t59get
Figure 2: Real-world validation on wheel-legged robot
In the real-world experiments H2O manifests overwhelmingly better transferability on both standing still and moving straight tasks. In the standing still task, as demonstrated the comparative curves in Figure 2(c), the robot with H2O policy remains steady after 11 seconds (s), while CQL bumps into the ground and goes out control in 12s. DARC, DARC+ and SAC are not able to keep the robot balanced and quickly fail after initialization. In the moving straight task demonstrated in Figure 2()). H2O achieves great control performance to keep balanced and closely follow the target velocity 0.2m/s, while CQL exceeds by fairly large margin, nearly doubling the desired target velocity. Additionally, the angle 0 illustrates that the robot controlled by H2O runs more smoothly than CQL. Once again, DARC, DARC+ and SAC fail the beginning. Interestingly, evaluated average returns both simulation and the real-world robot for SAC and H2O, revealing that achieving high performance in the simulation environment provides no guarantee for real-world transferability, if the simulator has a large sim-to-real gap. It is possible that policies with lower performance in the biased simulation could perform better real-world scenarios. This suggests the potential need of reconsidering the widely adopted simulation based evaluation or verification in cession-critica tasks, such autonomous driving and industrial control, that overly trusting simulators with dynamics gaps can lead to serious consequences.
plot the
6.4 Ablation Study
In this section, we investigate the impact of each design component of H2O in the HalfCheetah- Medium Replay-Gravity task. In Table 2, we compare the performances the variants of H2O by replacing adaptive value regularization weight w into uniform value, removing the importance weight in modified Bellman Error formulation E even removing the entire value regularization part from the framework. Without the adaptive value regularization, "H20-a" demonstrates lower performance since the Q-values at high dynamics-gap samples are not properly regularized during training. Without the dynamics ratio as importance weight to fix the Bellman error, "H20-dr", "H20-a-dr" and "H20- reg-dr" shows severe performance deterioration against their calibrated counterparts, which signifies the importance of correctly handling the Bellman updates when involving multiple sources of data. Removing the entire value regularization part from H2O leads substantial performance degradation. However, it is noteworthy that "H20-reg-drf outperforms "H20-dr", again suggests that fixing the Bellman error due to dynamics gap is very important in hybrid offline-and-onlin policy learning.
To further investigate the quality of the dynamics gap measure u(s, a) in H2O well its impact on the adaptive value regularization during the learning process, we construct a new test environment based on HalfCheetah. We apply specific noise offsets to actions based on X- X-velocity (see Figure 3) in the simulation, such that larger X-velocity bears larger noise offsets. We plot the dynamics gap measure us, a) and the corresponding Q-values Q( a) for each simulated sample (s, a) from
Please refer to supplementary video for performances of H2O and baselines the real wheel-legged robot.

---

Table Ablations on H2O. "-a" denotes no adaptive weights w. "-dr" denotes no dynamics ratio to fix the Bellman error. "-reg" stands for no value regularization applied in the framework.
H20
H20
420-dr
a2
H20-reg
120-
CQI
SAC
Adaptive Uniform
x
Modified 8 (Q.8*
X
Average Return
6813.1289
179
4721
5223
198
6501 1 147
$290
2774
45131111
(a)
epoch
(b) 20 epochs
(c) 200 spachs
(4) see epochs
Notie
-0.05
-0.10
-0.15
-0.20
44-
X-Weecty
XWeeectt
XANiecky
Figure 3: The dynamics gap measure u(s, a) evaluated during the training process
batch in some stages of the learning process. Notably, in the HalfCheetah task, higher X-velocity corresponds higher rewards and hence corresponds to higher Q-values, which at the same time may suffer from more severe dynamics gaps. We observe that as the course of learning proceeds, the dynamics gap measures in H2O indeed capture the higher dynamics gap information at the later part of the training, and works as expected to penalize the Q-values on high dynamics- gap samples.
7 Conclusion and Perspectives
In this paper, we propose the dynamics-aware Hybrid Offline-and-Online RL (H20) framework to combine offline and online RL policy learning, while simultaneously addressing the sim-to-real dynamics gaps in imperfect simulator. H2O introduces a dynamics-aware policy evaluation scheme, which adaptively penalizes the Q-values as well as fixes the Bellman error on simulated samples with large dynamics gaps. This scheme can be shown as equivalent to special kind of reward adjustment, which places reward penalties on high dynamics-gap samples, but boosts reward in regions with low dynamics gaps and abundant offline data. Under H2O, both the offline dataset and the imperfect simulator are fully used and complement each other to extract the maximum information for policy learning. In particular, the limited size and state-action space coverage of the offline datasets are greatly supplemented by unrestricted simulated samples; the dynamics gaps in simulated samples can also be alleviated during policy learning given correct dynamics information from the real data. Through extensive simulation and real-world experiments, we demonstrate the superior performance of H2O against other cross-domain online and offline RI. methods.
Online and offline RL have been two separately studied fields in the past, and both face a set of specific challenges. Few attempts have been made to organically combine these two different RL paradigms. H2O provides brand new hybrid offline-and-onii when leveraging offline data and simulation-basee there are several future works that can be done further enhance the model performance, such as adopting a less conservative offline RI. backbone algorithm as compared to CQL,, and incorporating better dynamics gap quantification methods that do not need approximation. We hope the insights developed in this work can shed light on future hybrid RL algorithm design, which can potentially provide better RL solutions for practical deployment.
RL paradigm, which shows promising results online learning in an integrated framework. Still,
Acknowledgments and Disclosure of Funding
This work supported by funding from Ha Haoyi Niu is also funded by Tsinghua Under- graduate "Future Scholar" Scientific Research Grant, i.e. Tsinghua University Initiative Scientific Research Program (2021 70200007. The authors would also like to thank the anonymous reviewers for their feedback on the manuscripts.
10

---

References
Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-21) 20, 2020. 3
Andrew G Barto, Richard S Sutton, and Charles W Anderson. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE transactions on systems, man, and cybernetics, 846,, 1983. 3
Dimitris Bertsimas, David B Brown, and Constantine Caramanis, Theory and applications of robust optimization. SIAM review, 53(3):464-555 501, 2011. 16
Alex Bewley, Jessica Rigley, Yuxuan Liu, Jeffrey Hawke, Richard Shen, Vinh-Dieu Lam, and Alex Kendall. Learning to drive from simulation without real world labels. In 2019 International conference on robotics and automation (ICRA), pages 4818-4824 IEEE, 2019. 2
Jacob Buckman, Carles Gelada, and Mare G Bellemare. The importance of pessimism in fixed-dataset policy optimization. In International Conference on Learning Representations, 2020.
Yevgen Chebotar, Ankur Handa, Viktor Makoviychuk, Miles Macklin, Jan Issac, Nathan Ratliff, and Dieter Fox. Closing the sim-to-real loop: Adapting simulation randomization with real world experience. In 2019 International Conference on Roboties and Automation (ICRA), pages 8973-8979. IEEE, 2019.
Benjamin Eysenbach, Shreyas Chaudhari, Swapnil Asawa, Sergey Levine, and Ruslan Salakhutdinov. Off-dynamics reinforcement learning: Training for transfer with domain classifiers. In International Conference on Learning Representations, 2020. 2, 3, 5, 8, 19, 24
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.072/9, 2020. 7
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach offline reinforcement learning. Advances in Neural Information Processing Systems, 34, 2021. 2, 3
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International Conference Machine Learning, pages 1587-1596, 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pages 2052-2062. PMLR, 2019. 2,3
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International Conference on Machine Learning, pages 1861-1870, 2018. 5,
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2020.
Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning, pages 5783. PMLR, 2021. 2.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In International Conference Learning Representations, 2022. 2,
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances Neural Information Processing Systems, pages 11761-11771, 2019. 2, 3.4
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Newral Information Processing Systems (NeurIPS), 2020. 2, 3, 4, 7, 19, 20

---

Alex X Lee. Coline Manon Devin, Yuxiang Zhou, Thomas Lampe, Konstantinos Bousmalis, Jost To- bias Springenberg. Arunkumar Byravan, Abbas Abdolmaleki. Nimrod Gileadi, David Khosid, et al. Beyond pick-and-place: Tackling robotic stacking of diverse shapes. In 5th Annual Conference on Robot Learning. 2021.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint 200.i::20000006443 2020. 2, 3
Jianxiong Li, Xianyuan Zhan, Haoran Xu, Xiangyu Zhu, Jingjing Liu, and Ya-Qin Zhang, Distance- sensitive offline reinforcement learning. arXiv preprint arXiv:2205.10027, 2022.
Jinning Li, Chen Tang, Masayoshi Tomizuka, and Wei Zhan. Dealing with the unknown: Pessimistic offline reinforcement learning. In Conference on Robot Learning, pages 1455-1464, PMIR, 2022.
3, 4. 19
JG Liao and Arthur Berg. Sharpening jensen's inequality. The American Statistician, 2018. 6, 16
Jinxin Liu, Zhang Hongyin, and Donglin Wang. Dara: Dynamics-aware reward augmentation in offline reinforcement learning. In International Conference on Learning Representations, 2022.
Lennart Ljung. System identification. In Signal analysis and prediction. pages 163-173. Springer, 1998.
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:2 10470, 2021.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 112.v:131 2013.
Xue Bin Peng, Marcin Andrychowicz, Wojeiech Zaremba, and Picter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 20/8 IEEE international conference robotics and automation (ICRA), pages 3803-3810. IEEE, 2018. 2,3
Aravind Rajeswaran, Sarvjeet Ghotra, Balaraman Ravindran, and Sergey Levine. Epopt: Learning robust neural network policies using model ensembles. arXiv preprint arXi::161000128; 2016.
Kanishka Rao, Chris Harris, Alex Irpan, Sergey Levine, Julian Ibarz, and Mohi Khansari. RI-cyclegan: Reinforcement learning aware simulation-to-real. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11 11157-11166 2020. 2
David Silver, Julian Schrittwieser, Karen Simonyan, loannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354-399 2017.
Richard S Sutton, Andrew G Barto, et al. Introduction to reinforcement learning. 1998. 3
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wajciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 JEEE/RSJ international conference on intelligent robots and systems (IROS), pages 23-30. IEEE, 2017.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033. IEEE, 2012.
Quan Vuong. Sharad Vikram, Hao Su, Sicun Gao, and Henrik I Christensen. How to pick the domain randomization parameters for sim-to-real transfer of reinforcement learning policies? arXiv preprint
2019. 3
Guan Wang, Haoyi Niu, Desheng Zhu, Jianming Hu, Xianyuan Zhan, and Guyue Zhou. A versatile and efficient reinforcement learning approach for autonomous driving. Machine Learning for Autonomous Driving Workshop NeurlPS 2022, 2022. 2
12

---

Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:191..11366 2019. 3
Teng Xiao and Donglin Wang. A general offline reinforcement learning framework for interactive recommendation. In The Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI, volume
2021, 2021. 2
Haoran Xu, Xianyuan Zhan, Jianxiong Li, and Honglei Yin. Offline reinforcement learning with soft behavior regularization. arXiv preprint arXiv:21 2021. 3
Haoran Xu, Jiang Li, Jianxiong Li, and Xianyuan Zhan. policy-guided imitation approach for
offline reinforcement learning. In Advances in Neural Information Processing Systems, 2022. 2, 3
Haoran Xu, Xianyuan Zhan, and Xiangyu Zhu. Constraints penalized q-learning for safe offline reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2022.
Wenhao Yu, Jie Tan, C Karen Liu, and Greg Turk. Preparing for the unknown: Learning universal policy with online system identification. arXiv preprint arXiv:1702.02453, 2017. 3
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. In Neural Information Processing Systems (NeurIPS), 2020. 3, 23
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. Advances in Neural Information Processing Systems, 34, 2021. 3, 4, 19, 23
Xianyuan Zhan, Haoran Xu, Yue Zhang, Xiangyu Zhu, Honglei Yin, and Yu Zheng. Deepthermal: Combustion optimization for thermal power generating units using offline reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2022. 2.3
Xianyuan Zhan, Xiangyu Zhu, and Haoran Xu. Model-based offline planning with trajectory pruning. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022, pages 3716-3722, 2022. 3
13

---

Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 7, Conclusions and Perspectives
(C) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section 4, and Appendix A
(b) Did you include complete proofs of all theoretical results? [Yes] See Section 5 and Appendix A
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix B.1
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [Yes] See Section 6
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix B.1
4. If you using existing assets (e.g., code, data, models) or curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent obtained from people whose data you' using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "Jdy1BQSXWfo", "reviewer": "Reviewer_hCpT", "summary": "This paper is based on the premise that we have an imperfect simulator for online RL, as well as a fixed dataset for offline RL. It proposes a new policy evaluation objective that is designed to penalize the Q values at (s,a) pairs where there is a big gap between simulation and real. The gap is measured by two discriminators trained following DARC.\n\nWith manually modified environment dynamics on MuJoco-HalfCheetah task as well as a real robot experiment, the authors showed that their algorithm is able to better utilize both the imperfect simulator and the limited but accurate offline data. \n", "questions": "* (Apologies if those details are already mentioned in the appendix.)\n* What is the distribution of the human performance on the collected dataset for the real robot?\n* For l173, the exact closed form solution I got is d = w(s,a) exp (Q(s,a) - 1). Can you double check if that is correct, and comment on whether it is preferable to use the exact solution instead of the proportional one in l173? \n* In l184-185, what is the ratio r of experience from source vs. target that you used for the DARC like training? And did you try to tune that value? \n\nComments:\n* In Table 2, it is interesting that H2O-a is not that bad, indicating that perhaps the environment deviation is not really dependent on (s,a). It’d be interesting to look at environments where the sim-real gap is uneven across (s,a) pairs, e.g. the dead-zones mentioned in l282-283.\n\ntypos/nits:\n* L55: incorporating -> incorporate\n* L56: only restricted -> are only restricted\n* l296 : which does not impacted -> which is not impacted", "limitations": "The requirement of training two discriminators constrains the proposed method’s application to domains where enough offline dataset is available for training a discriminator. \n\nThe approximation of the expected value in l207 can be undefined for s’ unvisited in the offline dataset.\n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n* The paper is well written, especially the introduction and section 4, with clear steps and explicitly listed assumptions. \n* The authors did experiments both in the real and the simulated environments, validating their main claim that this method helps bridge the sim2real gap and makes RL more applicable in the real world.\n* The method proposed is original as far as the reviewer can tell.\n\nWeaknesses:\n* Some parts of the paper can benefit from a bit more clarification. See the Questions section below. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "FgGPXfewRnp", "reviewer": "Reviewer_X8Vb", "summary": "This paper proposes a novel algorithm of offline-to-online reinforcement learning for *domain-shift* between offline datasets and online executable environments. This might be an important setting for addressing real-world tasks, since we could typically access a limited amount of real-world data and easily accessible simulators. Since the source online executable simulators and the target offline data have a large dynamics gap, the proposed method, H2O, employs (1) adaptive value regularization for high dynamics-gap transitions and (2) dynamics-corrected Bellman error. These regularizations on the objective functions correspond to penalty rewards that punish highly dynamics-gapped samples in a tablular case. The experiments show transfers from the offline dataset on a normal simulator to unreal dynamics simulator in MuJoCo HalfCheetah, and a sim-to-real transfer with the wheel-legged robot.", "questions": "- Checklist 2 (a) is required to be fixed.\n- $\\beta$ in Eq 4 seems duplicated to a parameter of behavioral policy $\\pi_{\\beta}$.\n- TD3-BC (Fujimoto et al. [5]) might be a competitive baseline (BC with offline data, and optimizing TD3 term with simulator transitions). \n- Also, an offline model-based method such as MOPO may deal with this setting, by training the dynamics model with real-world offline data and training the policy within imaginary rollouts with a pessimistic penalty. \n- Citation of Liu et al. [2] seems to be presented in ICLR 2022, not ICLR2021.\n\n---\n[4] Fujimoto et al. A Minimalist Approach to Offline Reinforcement Learning. 2021.\n", "limitations": "- The proposed method seems to highly depend on CQL, compared to DARA [2].\n- The empirical evaluation is only done on HalfCheetah, which seems very limited, comapred to prior works [1, 2].\n", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[S] ... Strengths, [W] ... Weaknesses\n\n### Originality\n- [S] Offline-to-online reinforcement learning for domain-shift has not been explored yet, but seems an important problem for real-world applications.\n- [W] H2O seems highly co-dependent to CQL and DARC (Eysenbach et al. [1]), while relevant work in offline RL, DARA (Liu et al. [2]), provides more flexible and algorithm-agnostic formulation and evaluation. The difference between CQL and H2O is dynamics-gap-based reweighting $\\omega(s, a)$ compared to CQL(\\mathcal{H}), which uses uniform action distribution for the weighted sum of $\\exp(Q)$.\n\n### Quality\n- [S] This paper is well-structured and easy to follow.\n- [W] The experimental coverage might be limited because H2O is only tested on HalfCheetah (Table 1).\n- [W] I'm a bit confused about experimental settings on the simulator, described in Section 6.1. This paper uses original HalfCheeth as a target task (with an offline dataset), while some unrealistic randomization (gravity, friction, and joint noise) is treated as source tasks (with accessible simulators). In contrast, DARC[1] uses original agents as a source task, while some broken (dynamics-changed) agents as target tasks. I'm not sure why the authors flip the settings, and whether H2O works in a reverse setting. In my opinion, even if we could not model the real-world dynamics perfectly, we might use the best-efforted simulator for training. So the setting in DARC[1] seems more likely than that of this paper in the real-world problem.\n\n### Clarity\n- [W] I'm not sure whether density ratio estimation (via discriminators) in Eq 7 is needed or not, since H2O directly models transition dynamics from \"real\" offline datasets (Section 4.4) $P_{\\mathcal{\\hat{M}}}$, with Gaussian distribution as typical model-based RL method did. So it seems a straightforward but effective approach to model simulator \"source\" dynamics $P_{\\mathcal{M}}$ with Gaussian distribution.\n- [W] As for Figure 1 (Average Reward Bar graph), it might be valuable to include the results of other baselines, such as CQL and DARC(+). As far as I checked Appendix, there are no additional details.\n\n### Significance\n- [S] This work shows good sim-to-real finetuning results between IssacGym and real wheel-legged robots, which might be valuable to prove the proposed method might be directly connected to real-world application.\n- [W] I'm not fully sure that this work would be properly placed following previous sim-to-real literature. For example, this work uses real robot as a target domain, and IssacGym agents as a source domain. Since IsaacGym has a well-designed dynamics-parameter-randomization [3], and such simple parameter-randomized training helps the agent to successfully deal with real-world dynamics [4]. It might be beneficial to include such dynamics randomization as a baseline.\n- [W] While the authors say, \"domain randomization often needs manually-specified randomized parameters and nuanced randomization distributions (L77, 78)\", the choice of $\\omega(s,a)$ in H2O is also a heuristic. So randomized dynamics training might be a fair comparison against H2O.\n\n---\n[1] Eysenbach et al. Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifiers. 2021.\n\n[2] Liu et al. DARA: Dynamics-Aware Reward Augmentation in Offline Reinforcement Learning. 2022.\n\n[3] Makoviychuk et al. Isaac Gym: High Performance GPU Based Physics Simulation For Robot Learning. 2021.\n\n[4] Chen et al. A System for General In-Hand Object Re-Orientation. 2021.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "COdbGkevcqu", "reviewer": "Reviewer_eEjG", "summary": "This paper presents a method for learning from a combination of offline data collected for example in simulation where the dynamics may not necessarily be accurate, as well as data from the real-world with the “true” dynamics. The method is built off of the CQL offline RL algorithm and proposes a few techniques for incorporating additional offline data, including adaptive value regularization based on estimated dynamics gaps from learned classifiers, and reweighting the bellman error based on the shifting dynamics. The method is evaluated on simulated MuJoCo environments with modified dynamics, as well as a real world experiment using a wheeled robot to perform a balancing task. \n", "questions": "- It is surprising to me that the performance of H2O-a is very similar to that of H2O. My understanding is that H2O replaces the learned $\\omega(s, a)$ with a fixed, uniform $\\omega(s, a)$. But, doesn’t this remove the capability for the algorithm to detect any off-dynamics transitions at all? Doesn’t the dynamics ratio to fix the Bellman error “-dr” also depend on the learned $\\omega$? If so, why doesn’t it have the same performance as H2O-a-dr? Some clarification would be appreciated.\n\nNitpicks/typos:\n- L128: the notation for the behavior policy is introduced but it doesn’t seem like it’s actually used in equation 3? \n- L141: sate-action -> state-action\n- L165: the way the maximization term is written is not consistent with Eq. 4\n- L173: close-form -> closed-form\n- Fig 2: epoches -> epochs", "limitations": "The limitations of the work are not described in detail in the last section, although the authors provide a few potential future directions. I would encourage the authors to more explicitly address the limitations of the current work as well as potential negative societal impacts.\n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The problem of learning from real, offline data as well as a potentially inaccurate simulator is important and applicable, and to my knowledge, relatively unexplored. This work is a nice step towards being able to learn in this setting.\n- The formulation of the H2O objective is well-motivated and has a useful interpretation in terms of “confidences” in each state-action pair, depending on whether they are in-distribution for the offline data or have high dynamics gaps. The analysis in section 5 is quite nice.\n- The experimental results are strong, and particularly the real-world wheeled robot experiment helps to convince me that this method is able to handle in-the-wild situations where we have access to some real offline data and a simulator. \n\nWeaknesses: \n- Mostly, I am confused about the conclusions drawn from the ablation study. Based on the ablation study, it seems that the performance improvements of H2O over CQL in the simulation experiments are largely due to the reweighting of the Bellman updates rather than the value regularization part of the framework. I disagree with the statement that “Removing the entire value regularization part from H2O leads to substantial performance degradation”, as the performance of H2O vs H2O-reg in the table is within the standard error range. \n- The overall clarity of the writing could be improved. Below I note some specific points but I think it could be helpful to have some additional proofreading. \n\nClarity: \n- I think the clarity of section 6.1, detailing the experimental setups in simulation, could be improved. Specifically, I think it would help to clarify that the original MuJoCo environments are being considered the “real” environments, and the offline dataset described in L264 serves as offline data from the real world. \n- I don’t quite understand from reading the main paper how the objective derived in Section 4 is incorporated into the main algorithm. The algorithm box in appendix B.1 helps significantly, but I feel it would help to mention that this objective is being used during online learning. \n- The introduction of $\\omega$ in section 4.2 is rather confusing – it’s not described how $\\omega$ characterizes the dynamics gaps for the samples until later. I might recommend providing the KL divergence definition of $\\omega$ earlier or reformatting the writing slightly. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept