Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper is based on the premise that we have an imperfect simulator for online RL, as well as a fixed dataset for offline RL. It proposes a new policy evaluation objective that is designed to penalize the Q values at (s,a) pairs where there is a big gap between simulation and real. The gap is measured by two discriminators trained following DARC.

With manually modified environment dynamics on MuJoco-HalfCheetah task as well as a real robot experiment, the authors showed that their algorithm is able to better utilize both the imperfect simulator and the limited but accurate offline data. 

strengths_and_weaknesses: Strengths: 
* The paper is well written, especially the introduction and section 4, with clear steps and explicitly listed assumptions. 
* The authors did experiments both in the real and the simulated environments, validating their main claim that this method helps bridge the sim2real gap and makes RL more applicable in the real world.
* The method proposed is original as far as the reviewer can tell.

Weaknesses:
* Some parts of the paper can benefit from a bit more clarification. See the Questions section below. 

questions: * (Apologies if those details are already mentioned in the appendix.)
* What is the distribution of the human performance on the collected dataset for the real robot?
* For l173, the exact closed form solution I got is d = w(s,a) exp (Q(s,a) - 1). Can you double check if that is correct, and comment on whether it is preferable to use the exact solution instead of the proportional one in l173? 
* In l184-185, what is the ratio r of experience from source vs. target that you used for the DARC like training? And did you try to tune that value? 

Comments:
* In Table 2, it is interesting that H2O-a is not that bad, indicating that perhaps the environment deviation is not really dependent on (s,a). It’d be interesting to look at environments where the sim-real gap is uneven across (s,a) pairs, e.g. the dead-zones mentioned in l282-283.

typos/nits:
* L55: incorporating -> incorporate
* L56: only restricted -> are only restricted
* l296 : which does not impacted -> which is not impacted
limitations: The requirement of training two discriminators constrains the proposed method’s application to domains where enough offline dataset is available for training a discriminator. 

The approximation of the expected value in l207 can be undefined for s’ unvisited in the offline dataset.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents a method for learning from a combination of offline data collected for example in simulation where the dynamics may not necessarily be accurate, as well as data from the real-world with the “true” dynamics. The method is built off of the CQL offline RL algorithm and proposes a few techniques for incorporating additional offline data, including adaptive value regularization based on estimated dynamics gaps from learned classifiers, and reweighting the bellman error based on the shifting dynamics. The method is evaluated on simulated MuJoCo environments with modified dynamics, as well as a real world experiment using a wheeled robot to perform a balancing task. 

strengths_and_weaknesses: Strengths:
- The problem of learning from real, offline data as well as a potentially inaccurate simulator is important and applicable, and to my knowledge, relatively unexplored. This work is a nice step towards being able to learn in this setting.
- The formulation of the H2O objective is well-motivated and has a useful interpretation in terms of “confidences” in each state-action pair, depending on whether they are in-distribution for the offline data or have high dynamics gaps. The analysis in section 5 is quite nice.
- The experimental results are strong, and particularly the real-world wheeled robot experiment helps to convince me that this method is able to handle in-the-wild situations where we have access to some real offline data and a simulator. 

Weaknesses: 
- Mostly, I am confused about the conclusions drawn from the ablation study. Based on the ablation study, it seems that the performance improvements of H2O over CQL in the simulation experiments are largely due to the reweighting of the Bellman updates rather than the value regularization part of the framework. I disagree with the statement that “Removing the entire value regularization part from H2O leads to substantial performance degradation”, as the performance of H2O vs H2O-reg in the table is within the standard error range. 
- The overall clarity of the writing could be improved. Below I note some specific points but I think it could be helpful to have some additional proofreading. 

Clarity: 
- I think the clarity of section 6.1, detailing the experimental setups in simulation, could be improved. Specifically, I think it would help to clarify that the original MuJoCo environments are being considered the “real” environments, and the offline dataset described in L264 serves as offline data from the real world. 
- I don’t quite understand from reading the main paper how the objective derived in Section 4 is incorporated into the main algorithm. The algorithm box in appendix B.1 helps significantly, but I feel it would help to mention that this objective is being used during online learning. 
- The introduction of $\omega$ in section 4.2 is rather confusing – it’s not described how $\omega$ characterizes the dynamics gaps for the samples until later. I might recommend providing the KL divergence definition of $\omega$ earlier or reformatting the writing slightly. 

questions: - It is surprising to me that the performance of H2O-a is very similar to that of H2O. My understanding is that H2O replaces the learned $\omega(s, a)$ with a fixed, uniform $\omega(s, a)$. But, doesn’t this remove the capability for the algorithm to detect any off-dynamics transitions at all? Doesn’t the dynamics ratio to fix the Bellman error “-dr” also depend on the learned $\omega$? If so, why doesn’t it have the same performance as H2O-a-dr? Some clarification would be appreciated.

Nitpicks/typos:
- L128: the notation for the behavior policy is introduced but it doesn’t seem like it’s actually used in equation 3? 
- L141: sate-action -> state-action
- L165: the way the maximization term is written is not consistent with Eq. 4
- L173: close-form -> closed-form
- Fig 2: epoches -> epochs
limitations: The limitations of the work are not described in detail in the last section, although the authors provide a few potential future directions. I would encourage the authors to more explicitly address the limitations of the current work as well as potential negative societal impacts.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes a novel algorithm of offline-to-online reinforcement learning for *domain-shift* between offline datasets and online executable environments. This might be an important setting for addressing real-world tasks, since we could typically access a limited amount of real-world data and easily accessible simulators. Since the source online executable simulators and the target offline data have a large dynamics gap, the proposed method, H2O, employs (1) adaptive value regularization for high dynamics-gap transitions and (2) dynamics-corrected Bellman error. These regularizations on the objective functions correspond to penalty rewards that punish highly dynamics-gapped samples in a tablular case. The experiments show transfers from the offline dataset on a normal simulator to unreal dynamics simulator in MuJoCo HalfCheetah, and a sim-to-real transfer with the wheel-legged robot.
strengths_and_weaknesses: [S] ... Strengths, [W] ... Weaknesses

### Originality
- [S] Offline-to-online reinforcement learning for domain-shift has not been explored yet, but seems an important problem for real-world applications.
- [W] H2O seems highly co-dependent to CQL and DARC (Eysenbach et al. [1]), while relevant work in offline RL, DARA (Liu et al. [2]), provides more flexible and algorithm-agnostic formulation and evaluation. The difference between CQL and H2O is dynamics-gap-based reweighting $\omega(s, a)$ compared to CQL(\mathcal{H}), which uses uniform action distribution for the weighted sum of $\exp(Q)$.

### Quality
- [S] This paper is well-structured and easy to follow.
- [W] The experimental coverage might be limited because H2O is only tested on HalfCheetah (Table 1).
- [W] I'm a bit confused about experimental settings on the simulator, described in Section 6.1. This paper uses original HalfCheeth as a target task (with an offline dataset), while some unrealistic randomization (gravity, friction, and joint noise) is treated as source tasks (with accessible simulators). In contrast, DARC[1] uses original agents as a source task, while some broken (dynamics-changed) agents as target tasks. I'm not sure why the authors flip the settings, and whether H2O works in a reverse setting. In my opinion, even if we could not model the real-world dynamics perfectly, we might use the best-efforted simulator for training. So the setting in DARC[1] seems more likely than that of this paper in the real-world problem.

### Clarity
- [W] I'm not sure whether density ratio estimation (via discriminators) in Eq 7 is needed or not, since H2O directly models transition dynamics from "real" offline datasets (Section 4.4) $P_{\mathcal{\hat{M}}}$, with Gaussian distribution as typical model-based RL method did. So it seems a straightforward but effective approach to model simulator "source" dynamics $P_{\mathcal{M}}$ with Gaussian distribution.
- [W] As for Figure 1 (Average Reward Bar graph), it might be valuable to include the results of other baselines, such as CQL and DARC(+). As far as I checked Appendix, there are no additional details.

### Significance
- [S] This work shows good sim-to-real finetuning results between IssacGym and real wheel-legged robots, which might be valuable to prove the proposed method might be directly connected to real-world application.
- [W] I'm not fully sure that this work would be properly placed following previous sim-to-real literature. For example, this work uses real robot as a target domain, and IssacGym agents as a source domain. Since IsaacGym has a well-designed dynamics-parameter-randomization [3], and such simple parameter-randomized training helps the agent to successfully deal with real-world dynamics [4]. It might be beneficial to include such dynamics randomization as a baseline.
- [W] While the authors say, "domain randomization often needs manually-specified randomized parameters and nuanced randomization distributions (L77, 78)", the choice of $\omega(s,a)$ in H2O is also a heuristic. So randomized dynamics training might be a fair comparison against H2O.

---
[1] Eysenbach et al. Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifiers. 2021.

[2] Liu et al. DARA: Dynamics-Aware Reward Augmentation in Offline Reinforcement Learning. 2022.

[3] Makoviychuk et al. Isaac Gym: High Performance GPU Based Physics Simulation For Robot Learning. 2021.

[4] Chen et al. A System for General In-Hand Object Re-Orientation. 2021.

questions: - Checklist 2 (a) is required to be fixed.
- $\beta$ in Eq 4 seems duplicated to a parameter of behavioral policy $\pi_{\beta}$.
- TD3-BC (Fujimoto et al. [5]) might be a competitive baseline (BC with offline data, and optimizing TD3 term with simulator transitions). 
- Also, an offline model-based method such as MOPO may deal with this setting, by training the dynamics model with real-world offline data and training the policy within imaginary rollouts with a pessimistic penalty. 
- Citation of Liu et al. [2] seems to be presented in ICLR 2022, not ICLR2021.

---
[4] Fujimoto et al. A Minimalist Approach to Offline Reinforcement Learning. 2021.

limitations: - The proposed method seems to highly depend on CQL, compared to DARA [2].
- The empirical evaluation is only done on HalfCheetah, which seems very limited, comapred to prior works [1, 2].
review_title: null

Meta-review: metareview: The authors present a novel but realistic problem, where you want to learn from limited offline real-world data, and from unlimited simulation data, as is generally the case for sim2real with real-world finetuning. Experiments are performed on D4RL HalfCheetah with three artificially created dynamics gaps, as well as real-world wheeled robot. 

Thanks to the reviewers and authors for engaging in active discussions in experimental details, and appreciate the authors for updating the draft with additional experiments with TD3+BC, Random_HalfCheetah etc. I recommend the paper for acceptance, as this paper could encourage further research into this important problem setting.

confidence: Certain
Decision: Accept