Targeted Environment Design from Offline Data
Authors: Izzeddin Gur
Ofir Nachum
Aleksandra Faust
Venue: ICLR 2022 Submitted (2022)
Keywords: targeted environment design
offline reinforcement learning
deep learning
adversarial learning

Abstract: In reinforcement learning (RL) the use of simulators is ubiquitous, allowing cheaper and safer agent training than training directly in the real target environment. However, this approach relies on the simulator being a sufficiently accurate reflection of the target environment, which is difficult to achieve in practice, resulting in the need to bridge sim2real gap. Accordingly, recent methods have proposed an alternative paradigm, utilizing offline datasets from the target environment to train an agent, avoiding online access to either the target or any simulated environment but leading to poor generalization outside the support of the offline data. We propose to combine the two paradigms: offline datasets and synthetic simulators, to reduce the sim2real gap by using limited offline data to train realistic simulators. We formalize our approach as offline targeted environment design(OTED), which automatically learns a distribution over simulator parameters to match a provided offline dataset, and then uses the learned simulator to train an RL agent in standard online fashion. We derive an objective for learning the simulator parameters which corresponds to minimizing a divergence between the target offline dataset and the state-action distribution induced by the simulator. We evaluate our method on standard offlineRL benchmarks and show that it learns using as few as 5 demonstrations, and yields up to 17 times higher score compared to strong existing offline RL, behavior cloning (BC), and domain randomization baseline, thus successfully leveraging both offline datasets and simulators for better RL

Under review as a conference paper at ICLR 2022
TARGETED ENVIRONMENT DESIGN FROM OFFLINE DATA
Anonymous authors Paper under double-blind review
ABSTRACT
In reinforcement learning (RL) the use of simulators is ubiquitous, allowing cheaper and safer agent training than training directly in the real target environment. How- ever, this approach relies on the simulator being a sufficiently accurate reflection of the target environment, which is difficult to achieve in practice, resulting in the need to bridge sim2real gap. Accordingly, recent methods have proposed an alter- native paradigm, utilizing offline datasets from the target environment to train an agent, avoiding online access to either the target or any simulated environment but leading to poor generalization outside the support of the offline data. We propose to combine the two paradigms: offline datasets and synthetic simulators, to reduce the sim2real gap by using limited offline data to train realistic simulators. We formalize our approach as offtine targeted environment design (OTED), which automatically learns distribution over simulator parameters match a provided offline dataset, and then uses the learned simulator to train an RL agent in standard online fashion. We derive objective for learning the simulator parameters which corresponds to minimizing a divergence between the target offline dataset and the state-action distribution induced by the simulator. We evaluate our method on standard offline RL benchmarks and show that it learns using as few as 5 demonstrations, and yields up 17 times higher score compared to strong existing offline RL., behavior cloning (BC), and domain randomization baseline, thus successfully leveraging both offline datasets and simulators for better RL.
INTRODUCTION
Recent years have witnessed the proliferation of reinforcement learning (RL) as a way to train agents to solve variety of tasks, encompassing game playing (Brown & Sandholm 2019
Silver et al.
2016) and some robotics applications (Akkaya al.) 2019; [Schulman et al. 2015: Chiang al. 2019), among others. These successes often rely on the use simulators, providing several benefits to online RL agent training: large amounts of low cost and more diverse data not readily available in reality (Tolani et al. 2021). and safe training that avoiding physical damage (Krishnan et al. 2021
human privacy concerns (Tolani et al. 2021). In many instances, for example in game playing, an accurate simulator is easy to devise. In others, rough approximations of the real target environment are sufficient (e.g. CChang et al. (2019) to close sim2real gap. However, in other applications such as robot manipulation (Kalashnikov al. 2018 Todorov et al., 2012). health (Ernst al., (006).
or web navigation (Gur et al.] (2021) designing the simulator to match a desired target environment
with sufficient fidelity can be a highly manual process (Nachum et al., 2019a Krishnan et al. 2021
potentially requiring repeated online access to the target environment (Chebotar et al. 2019 Ramos
al. 2019 Mehta al. 2020).
Due to the difficulty in matching simulators to target environments and motivated by the successes of supervised learning on large static datasets, a recent line of works Fujimoto et al. (2019); Agarwal et al.
(2020); Wu et al, (2019): Levine et al. (2020): Fu et al. (2020); Gulcchre et al. (020)) has proposed to circumvent online training altogether (whether on the simulator or in the target environment) and instead perform RL on offline dataset of logged experience in the target environment, typically collected by an unknown behavior policy. This approach avoids any potential mismatch between training and test environment. However, existing algorithms for offline RL are highly liable to generalization errors, and so these methods often impose a strong regularization on the learned policy to maintain proximity to the offline dataset (Fujimoto al. 2019 Wu et al. 2019 Kostrikov et al.

---

Under review as a conference paper at ICLR 2022
2021: Yu et al. 2020). Thus, such methods can rarely approach the performance of online RL on an accurate simulator.
Here, we propose to leverage both approximate simulators and offline datasets to train agents. Namely, we consider settings in which one has access to an offline dataset collected from a target environment as well approximate (but nevertheless inaccurate) simulator. How can we leverage both of these to learn a near-optimal policy for the target environment?
To tackle this problem, we propose an approach which, at high level, is composed of two stages: (1) use the offline dataset to learn a distribution over simulator parameters; (2) train an RL agent in an online fashion on the simulator using any off-the-shelf RL algorithm. The key question is then how to appropriately learn the simulator parameters from an offline dataset, which we term offline targeted environment design (OTED). For this problem, we devise an objective quantifying the divergence between the state-action distribution appearing in the offline dataset and the state-action distribution induced by a learned behavior policy acting in the simulator. We show that the simulator parameters and the target behavior policy can be jointly learned such that even suboptimal behavior policy, that would otherwise perform poorly on the target environment, can help learn simulator parameters very effectively. We further show how the performance of the final returned policy can be improved by employing OTED not only for policy training but also for offiine policy selection (Yang et al. 2020 Fu et al. 202Tp. Namely, we show how one may use the OTED- simulators to both provide set of candidate policies as well as evaluate those policies in order to rank and choose the best policy to return, reminiscent of cross-validation techniques popular in supervised learning Arlot & Celisse 2010) but so far elusive in RL contexts (Paine et al. 2020).
We apply our method to a variety of domains, including D4RL (uu et al. 2020) and MiniGrid (Chevalier-Boisver et al. 2018), showing that our method is able to recover the target environment parameter using as few also show that an online agent trained in designed simulators yields improved performance compared to methods which leverage only simulators or only offline datasets, as well existing methods for domain transfer TTobin et al. 2017). On the D4RL. (Fu et al. 2020) benchmark, we demonstrate that our method can even outperform an online SAC agent trained on the ground-truth target environment. Our method achieves up to 17 times higher score compared to previous state-of-the-art offline RL and behavior cloning (BC) methods as well as domain randomization. On datasets with medium-level behavior that can be collected without domain experts, show more relative improvements where our model even reaches the performance of the offline RL methods trained on expert-level behaviors.
demonstrations and learn from sigh-dimensional image observations. We
2 RELATED WORK
Our work focuses on learning simulator parameters to match a target environment and is thus related to large and diverse literature domain transfer in RL. One of the most common and simple
approaches to handling unknown simulator parameters is domain randomization (Sadeghi & Levine
2016
Tobin et al. 2017 Matas et al. 2018 Chiang et al. 2019 Tolani et al. 2021: Krishnan et al.
2021. which randomizes the setting of those unknown parameters during training. thus learning an
agent which is robust to wide distribution of possible simulators. While domain randomization has demonstrated successes, the process of choosing appropriate ranges for unknown parameters can be highly manual process (Andrychowicz et al., (020), in which one must balance between either too wide range that hampers agent training, and too narrow range that may exclude the target environment parameterization (Nachum al. 2019a). While some works have proposed heuristic mechanisms to better automate this process (Akkaya et al. 2019 Krishnan et al. 202T). others have explored incorporating online interaction with the environment into this tuning process (Chebotar et al. 2019: Ramos et al. 2019 Mehta et al., 2020). In this work, we avoid any online interactions with the target environment during training, and only assume access to a static offline dataset.
The proposed OTED method, which uses an offline dataset to learn a simulator, is related to recent model-based approaches to offline RL (Malsushima et al. 2020 Yu et al. 2020 Argenson & Auloc-Arnold 2020). In these works, the offline dataset learns approximate dynamics and reward models of the environment, and the agent optimizes behavior in this approximate model of the environment. Although this makes minimal assumptions the target environment, using reward and dynamics models as proxies for the environment leads to extrapolation issues which necessitate strong regularizers the learned policy. Moreover, these existing approaches mostly ignore the
2

---

Under review as a conference paper at ICLR 2022
fact that in many scenarios, one already has approximate simulator, albeit not differentiable; e.g.. the physical laws in a continuous control environments are straightforward to implement (Todorov) al. 2012). although parameters more specific to the environment (friitionn coefficient, actuator gain, gravity) are unknown. Leveraging these simulators and focusing learning on the unknown parameters, modelling the full dynamics from scratch.
as performed by our approach, can thus improve
Our proposed objective is based on a distribution matching loss. Similar losses have appeared in the past, especially in generative models (Goodfellow al. 2014), off-policy evaluation (Nachum
et al. (2019)) off-policy learning (Nachum et al. 2019c). and imitation learning (Ho & Ermon 2016]
Stadie et al. 2017: Kostrikov et al. 2020). These predominantly focus on learning a policy an adversarial objective and ignore the mismatch between simulators and real environments. Even when the distribution matching objective is optimal, this mismatch might give very suboptimal policies, due to potential poor support of the data distribution. This is very realistic scenario since tuning simulator parameters accurately a very tedious process and requires domain-experts
Finally, we emphasize that our proposed approach is distinct from previous work which suggests to combine offline learning on static datasets with online learning on the target environment (Nair
et al. 2020, Ajay et al., 2020 Yang & Nachum 202T Xie et al.) 202T Lee et al., 202T). Specifically, we avoid any online interactions with the target environment, and only allow online training on the simulator, which may may not be good approximation to the target environment.
3 BACKGROUND
We consider an RL setting in which we are given access to both an offline dataset of experience from the target environment and an approximate simulator, in which the agent can gather more experience. We elaborate on the notation and background relevant to this setting.
Reinforcement Learning We define an environment as a Markov Decision Process (MDP) given by a tuple (S, A. Pv. T.r, 7) where S is a state space, A is an action space, Po is an ini- tial state distribution, T(s' a) is state transition distribution, r : S A - I is reward function over state and actions, and E 1) is discount factor. A stationary policy in this environment is a function from states to distributions over actions. In reinforcement learn- ing, we are interested in finding policy that maximizes the cumulative discounted returns: 1 7'r(st, az)]ss ~ Po. ag ~ (5t), ~ T(s,, 4:)] We will use d' d'(ss,) to denote the state-action distribution of क: (= = (1 - 7Pr(s: = 8, = also ~ Pa, az ~ T (8:)), 86+1 T(s),
Simulator vs. Target Environment Typically in online reinforcement learning. policy is trained based experience collected by the agent in a simulator. While this implicitly assumes that the simulator can perfectly mimic the target environment, in practice this is often not the case. The simulator typically only provides an approximation to the true initial state distribution Pu and state transition distribution T. For example in robotics applications, while physics simulators can be built, it is infeasible to accurately set all appropriate parameters (c.g.. friction coefficients, actuator gains, wind speed). In this work, we assume such a parameterized simulator exists, and it is given by M(z) (S, A, po(- T 2)) 7). where denotes the parameters of the simulated environment. Naive approaches would either set z heuristically or using randomization methods. In this work, we aim to use offline datasets to learn z so that M(2) approximates the target MDP, which we denote as M - We use d= (s, a|2) to denote the acte-nction distribution of x in the parameterized simulator M(2).
Offline Reinforcement Learning Offline RL, as an alternative to online RL on simulators, assumes that an agent cannot collect any online experience and restricted to a dataset 1) of (s. a, s') collected by an unknown behavior policy μn; i.e., (s, a) ~ d s' ~ p(s, a). We use d' (s, a) to denote the distribution over state-action pairs appearing in D. When trained on a fixed dataset, previous state-of-
the-art online reinforcement learning algorithms such SAC (Haarnoja al. 2019) underperform due to extrapolation errors (Fujimoto et al. 2019: Kumar et al.) 2019). which stem from sparsity of the samples in the dataset. Accordingly, many RL algorithms designed specifically for the offline
setting make use of strong regularizations to maintain proximity to the offline dataset (Wu et al.
2019 Kostrikov et al.] 2021 Yu et al. 2020
3

---

Under review as a conference paper at ICLR 2022
Offline Dataset
Train
Discriminator
Train
Train
Online Dataset
Train
Designer
Train Designed Simulator
Design
Collect
Act
Act
Behavior
Simulated
Online Policy
Policy
Environment
Obs
Obs, Reward
Learn Simulator Parameters
The simulator parameters are learned by the designer, which uses signals from a learned discriminator to match the distribution appearing in the offline dataset that induced by a learned behavior policy acting in the environment. The learned simulator parameters are then used train a target policy via standard online RL.
Figure 1: Outline of OTED:
4 OTED: OFFLINE TARGETED ENVIRONMENT DESIGN
We now describe our method for leveraging both offline datasets and simulators to learn return- maximizing policies, thus alleviating both of the above issues: data sparsity and extrapolation issues when performing offline learning and (ii) inaccuracy of an approximate simulator for online learning.
At a high-level, our algorithm can be summarized as follows, also illustrated in Figure
Learn an approximate behavior policy á from D using any off-the-sheli imitation learning algorithm; c.g., behavioral cloning (Pomerleau 1991) ValueDICE (Kostrikov et al. 2020
2. Learn a distribution g(z). also called designer, over simulator parameters so that d (s, a/2) approximates the state-action distribution appearing in D.
3. Use any off-the- online RL algorithm e.g., SAC (Haarnoja al. 2019) learn a return-maxinn maximiiiiiiiin policy on the simulator M(z), where Z is sampled anew from the learned distribution ((2) the beginning of each episode.
Steps (1) and (3) above are straightforward, and step (2) below, and this is also summarized in Algorithm [ We then continue to show how we reduce variance inherent in the procedural three-step process above by using replicated experiments, distributions over mixtures of simulator parameters, and model selection without online access.
focus this section first elaborating
4.1 LEARNING SIMULATOR PARAMETERS VIA DISTRIBUTION MATCHING
Given an approximate behavior policy P. we propose to learn distribution q(2) over simulator parameters so that ds,s alz) approximates the state-action distribution appearing in D. Thus, our objective for q(=) based on a probability divergence between d (s(ssa) a/z) and dP a). encourage better generalization, we also introduce prior p(z), and measure divergence between dl (s, (2)) and (s,aa())(2)) Our proposed objective is given by.
d (s.
(RI(d' (s, (l2) (s,))((=)) =
E
[
log di 2 + DKzz((<))p(E)) (1)
(x))()) (e.a)rdd (s.als)
As desired, it is clear that the first term above encourages the parameter distribution g(2) match the state-action distribution of the approximate behavior policy j in the simulator to that of the dataset; meanwhile, the second term regularizes the simulator distribution from collapsing. We can
4

---

Under review as a conference paper at ICLR 2022
immediately see that the objective above is RL (actually, a bandit) objective for q, where the ("(s.a) actions are 2, the stochastic reward is given by log ["(ssa]s] for (s, a) ~ dd alz), and a relative entropy regularizer is applied to q. Therefore, one may use any off-the-shelf policy gradient algorithm (=.a)
to learn q, as long as one has access to the density ratios (=(eeo)))
However, in general one does not have access the densities dl dD explicitly. Nevertheless, one can easily sample from either density; samples from d are given by running the simulator M(z) and samples from dd are given by the dataset D itself. number of algorithms exist for estimating density ratios of two distributions given sampling access. In our implementation, we use the discriminative approach based on (gguyen al. 2010p. Namely, we train a discriminator g(s, a) to minimize the objective
(s (- log(1 1 ((s,a)))
(os.a) log log ()))))
similar to the discriminator in GAIL (Ho & Ermon 2016 When trained to optimality, the discrimi-
nator will satisfy the following, log 1-9" (s.a) = log (=(a,,,))) and this can be used as a reward signal for training q.
(e.0)
(a,a)
In practice, to allow for more flexibility in the trade-off between exploitation and regularization, we extend the objective with a tunable parameter A:
log d8(ss +[Dx)((()))(=)))
E
di di,ss as))
(e) (s.a)vvd (s,a/=)
small À encourages q to exploit the signal from the log-ratios, while a large À encourages to stay close to the prior p. In addition serving trade-off parameter, À can also be interpreted as (i) d (s.a)
reward scaling term in front of log ratios, i.e., log au(s,a]=] or (ii) a distribution smoothing term
applied to both d' and dd, i.e., log (c.e))
(d=(s,al=))
Similar to (Kostrikov et al. 2019). y, use samples stored in a replay buffer rather than online samples to improve sample efficiency. In practice, we adopt the commonly used approximation of log ratios: -log g(s,a) a)) instead of llog(1 (1 g(s,a) (g a)) in Eq.
Algorithm Training of the simulator parameters and the discriminator Input: Dataset D, initial behavior policy ü. initialized discriminator g, tunable simulator
Gather experience F, a sequence of (s, a) pairs, by running the policy f in the simulator S.
2: Estimate discriminator output, - -log(1 g(s, for each sample (s, a) in the experience E.
3: Update q(z) using the objective in Eq. 3 with the average - log(1 g(s a)) as the reward.
4: Update g(s, a) using the objective in Eq. with mini-batches sampled from (E, D).
5: Repeat above.
Return: (d(2),0(8,a))
Algorithm 2 Training of a single online agent using off-policy data and multiple simulator models
Input: Simulator distributions g(-lh;)) from n different experiments, simulator S, empty replay R
1: Sample i ~ 24[1, n² and set the simulator S using the mode 2, of the distribution g(-
2: Run the online agent [ sss in the simulator and populate the replay with new experience.
3: Sample a mini-batch from the replay and update the agent parameters.
4: Repeat above.
Return: (a]s)
5

---

Under review as a conference paper at ICLR 2022
Our objective. Eq. resembles mutual information when the joint distribution is defined using the proposal distribution and marginal distributions are defined using expert data distribution. The main difference is that marginals while in mutual information they are the same. Through optimizing the behavior policy to mimic the expert better, the objective will get closer mutual information formulation.
the behavior policy in the joint distribution and use expert policy in
4.2 REDUCING VARIANCE WITH REPLICATED EXPERIMENTS
Our proposed three step process approximate P., then learn (2), and finally training introduces noise at each step due to randomness in optimizers and learning algorithms. It is important to reduce the variance inherent in this process so that the returned policy is best prepared for evaluation in the target environment. We tackle this problem via two techniques: simulator selection and simulator-based policy selection.
Simulator selection As typical, we train the objective Eq. Jusing different hyper parameter settings where each setting will give different simulator model. Since the target parameter distribu- tions are unknown, this raises the question how do we choose the ideal set of hyperparameters? We propose to the first term in Eq. T namely expected log ratios, score and rank candidate simulators trained with different hyperparameters. We detail our procedure to select simulator hyperparameters in below steps:
Given a hyperparameter setting h, replicate it with n different seeds and train new OTED for each seed. This will output design distribution (-11,,), a discriminator g(-lh,), and an approximate behavior policy p(.]ha), where we use h2 to denote the use of hyperparameters à with seed i.
2. Run approximate behavior policies j(-14s) in their corresponding candidate simulators M(z) where ~ q(-1hk) and collect experience
3. Estimate discriminator outputs, c(h;) = T& E(sa)EE log(1 a(s,) score h,,
4. Score each hyperparameter h by averaging scores of their corresponding seeds, c(h) Σ c(hs).
5. Pick the hyperparameter with highest score e(h), corresponding to the simulator with the lower approximated KL divergence from the offline data.
Simulator- policy selection Even once specific simulator is chosen according to the above
procedure, training an online RI. agent on the simulator can result in drastically different policies, again due to randomness in the learning process. This presents the problem of offine policy selection,
which is a well-known challenge for real-world RL (Yang et al.) 2020 Paine et al. 2020 Fu et al. 2021). We propose simple approach to this, described in below steps:
1. Sample design parameters 2; (mode of the (-(lh)) for each seed from the selected hyperpa- rameter h in the above procedure.
2. Train m seeded on-policy agents n, on the distribution of simulators MMzz)) (see Algorithm
3. Evaluate the performance of policies my in each of the simulators M(2)).
4. Select the top performing policies according to their achieved the rewards averaged over all simulators.
One can view this offline evaluation as a form of rudimentary cross-validation, similar to existing techniques in offline policy selection which evaluate a learned policy according Q-values learned separate from the Q-values used to train the policy (Paine et al. 2020). Although the separately- trained Q values (like 7())) are trained on the same offline dataset, this simple approach has been demonstrated to achieve strong performance, even compared to more sophisticated methods (Fu et al.
2021

---

Under review as a conference paper at ICLR 2022
expert
CMM 20010
14060 10000
2000
1970
150000
90000 20000
2008
1900
11000 1012 1006 3000
(a) geom-friction
(b) actuator-gainprm
(c) gravity
(d) all parameters
Figure 2: Absolute error of OTED on different simulator parameters using the D4RL expert dataset.
We that OTED is able to accurately learn the true simulator parameters, using only access to suboptimal offline dataset.
5 EXPERIMENTS
5.1 EXPERIMENT SETUP
We conduct several key experiments to understand (i) OTED approximates target design distribu- tions well; if not, what factors contribute to its failure, (ii) if we can select good simulator model from large pool of candidates without knowing target design distribution, and (iii) if an online policy trained in simulator designed by OTED performs well across different environments.
We evaluate OTED, in a variety of settings; different tasks, data sources and simulator parameteriza- tions. We first evaluate our method on 3 continuous control tasks from the Mujoco [oooorov et al.] 2012) simulator, HalfCheetah, Hopper, and Walker2d, against state-of-the-art offline RL methods on the previous D4RL benchmark (Fu et al. 2020). D4RL datasets are divided into 3 levels, medium, medium-expert, and expert, reflecting the competency of the behavior policy used to collect the data, We modify the Mujoco simulator so that three 1 1-dimensional physics parameters geom-friction, actuator- gagitrrm, and gravity that cause different effects on the underlying simulator physics are unknown (See Appendix B for their ground truth values). We learn them both independently and jointly. Next, we experiment with a discrete MiniGrid (Chevalier-Boiswe) et al. 2018) environment to study if OTED can also learn from high-dimensional image observations. We parameterize a MiniGrid environment by the density of obstacles in a 10-by-10 grid.
We use SAC as the online agent specifically as it is not designed to work in the offline RL setting and call an online SAC agent trained in designed environments, OTED-SAC. We compare its performance to (i) previous state-of of- the-art offline RL methods, behavioral cloning (BC), F-BRC (Kostrikov al. 202T). MBOP (Argenson & Dulac-Arnold] 2020), (ii) online SAC agents trained with domain randomization (DR) (Tobin et al., (217)) and a baseline called interval search (IS) that is inspired by ADR where parameters are sampled from search interval that is gradually expanded over time based on the performance of an agent in the simulator, and (iii) a hybrid BC algorithm trained with both provided offline data and simulated data where the latter is collected using DR (BC+DR). We report (mean) absolute error to measure the error of final design predictions and ground truth designs. We also report online reward ground truth target environments for OTED-SAC and above baselines.
We compare our model selection method 2 baseline scoring metries: (i) design deviation, standard deviation of final design predictions across different seeds and (ii) learning progress, average log ratios collected during training. We pick simulator models with the lowest score for the former and highest for the latter. We report two metrics to evaluate model selection: (i) design error, difference between a design prediction and the ground truth design, and (ii) design regret, difference between the design errors of design prediction and the best prediction possible from the pool of available candidates.
We give more details to public upon publication. We also present experiments on alternative datasets taken from Kostrikov al. (2020) in the Appendix
model implementation in the Appendix A and will open source code
7

---

Under review as a conference paper at ICLR 2022
5.2 RESULTS ON D4RL BENCHMARK
We present results in three stages, first focusing the ability of our distribution matching objective to recover the true simulator parameters (Section 5.2.I). then presenting the results of the final learned policy according to OTED-SAC (Section 5.2.2 and finally performing ablation highlighting the importance of our model selection procedure (Section 5.2.3) 30
5.2. ACCURACY OF SIMULATOR PARAMETER PREDICTIONS
In Figure 2 we evaluate OTED on D4RL expert dataset using absolute error metric. We observe that OTED exhibits low error in learning every parameter both independently or jointly. This shows that OTED is capable of learning when parameters can interact in unknown and complicated ways. As an interesting use case, HalfCheetah emits a very high variance on the geom- crom-frictinn parameter early on but converges to smaller error. This is due to one experiment giving a much higher error and taking much longer to find the correct parameter rather than sensitivity of the HalfCheetah environment for this parameter. Upon inspection all parameter learning experiments, we observe that geom-friction is consistently the hardest parameter to learn and is the most prone to trapping a simulator model in a local optima. One particular reason is that, usually, the other two parameters are Iearned very quickly, allowing the discriminator to return high log ratios even before geom-friction is learned. In some experiments, this leads to simulator model following the inaccurate discriminator and converging into sub- optimal solutions. But, thanks to our simulator selection, we mitigate this issue and train online agents with accurate simulators.
We present more results with all expert levels in Appendix D
5.2.2 PERFORMANCE OF THE ONLINE AGENT WITH DESIGNED SIMULATORS
As OTED exhibits low absolute error on average, the natural question to ask is does this correlate well with the performance of an online agent trained on designed simulators? In Table we compare OTED-SAC to baseline RL algorithms using different expert levels. 100% performance in D4RL benchmark corresponds level expert datasets, OTED-SAC clearly outperforms previous RL. algorithms. This is particularly important as collecting medium-level demonstrations can be done effortlessly without an optimal policy. While previous algorithms fail when the offline dataset is collected via sub-optimal policy, OTED is still able to recover the unknown design distribution. Note that we also use BC to induce a behavior policy which is very sub-optimal across all medium-leve datasets. Even though Walker2D environments exhibit higher variance on geom-friction parameter for medium and medium-expent level experts, OTED-SAC outperforms previous models and also performs better compared to other parameterizations. In majority of the cases, OTED-SAC achieves more than 100% performance, outperforming a baseline SAC agent trained on the ground-truth target environment. Using offline and simulated data in BC algorithm (BC+DR), we get better performance than BC in medium and medium-expee expert datasets, particularly for Walker2D environments. Our method still outperforms this strong baseline, showing improvements beyond combining offline and simulated datasets.
SAC agent trained in the target environment. For medium-
5.2.3 SIMULATOR SELECTION
As presented above, success of an online agent is closely related to simulator being an accurate reflection of the target environment. As result, very critical to pick a simulator model from a large pool of candidates. In Table 2 compare simulator selection method to other baseline methods. Our proposed method outperforms baselines and picks the best model available in almost of cases (zero regret). Interestingly, design deviation is the second best metric and usually has low regret. We hypothesize the success of this metric to its ability to distinguish models with a number of different sub-optimal predictions; hence high deviation from the average prediction.
5.3 MINIGRID ENVIRONMENT RESULTS
We now move on MiniGrid, which presents us with mmae--baee tasks. We focus our evaluation on the simulator learned by OTED, showing that OTED is able to recover the true simulator parameters, even image-based environments.
8

---

Under review as a conference paper at ICLR 2022
OTED-SAC
BC
F-BRC
MBOP
DR
BC+DR
geom
actuator
friction
gainprm
gravity
all params
halfchectah- medium
36.1
41.3
44.6
6.7
12.2
36.6 (0.1)
(1.6)
106.7 (0.9)
118.8 (2.2) 116.6 (2.4)
35.8
93.3
105.9
6.7
12.2
50.6 (4.6)
80.0
109.3
113.6
114.2 (3.3)
halfcheetah-exper
107.0
108.4
6.7
12.2
105.7 (1.0)
119.0 (1.5)
109.6 (3.8)
110.5 (4.6) 116.3 (1.4)
hupper-medium
29.0
99.4
48.8
73.9
20.9
30.0 (0.1)
107.3 (5.8)
94.2 (8.6)
80.4 (9.9) 109.8 (2.6)
hopper-mediun-exye
111.9
112.4
55.1
73.9
20.9
53.0 (13.3)
114.0 (0.6)
98.5 (4.2)
109.6 (1.0) 81.9 (10.5)
hopper-expert
109.0
112.3
73.9
20.9
111.4
113.8 (1.1)
100.4 (5.0)
72.3 (18.8) 45.2
walker2dlmedium
6.6
78.8
41.0
46.2
26.9
16.0
113.4 (4.14 103.3 (12.9)
115.8 (3.7) 126.3 (3.6)
walker2d-modium--ee exppu
11.3
105.2
70.2
46.2
26.9
(3.2)
110.4 (1.9) 60.1 (18.6)
128.8 (2.9) 121.3 (2.8)
walker2d-cxpert
125.7
103.0
46.2
26.9
45.6 (6.9)
119.7 (2.7) 102.8 (3.7) 110.0 (1.8) 102.6 (2.5)
Table I: Performance comparison of our work (OTED-SAC) to existing RL algorithms on D4RL benchmark. We report offline RL algorithm results from (Kostrikov et al. 2021). OTED-SAC is able to recover expert policy across almost all settings. For geom-frictionn actuator-gainrrm
achieve on-par better than previous models. actuator- gainprm and gravity gives slightly lower performance than learning geom- foii-friition. We see almost no performance drop when all parameters trained jointly in medium-leve data.
Design Deviation
Learning Progress
Log Ratios (OTED)
haltchectah-expert
0.064 (0.003)
0.064 (0.003)
0.064 (0.003)
halfchectah- hattrraa--miimm---ee
0.061 (0.016)
0.061 (0.016)
0.061 (0.016)
halfcheetah- medium
0.073 (0.005)
0.071 (0.007)
0.071 (0.007)
hopper-expert
0.419 (0.024)
0.196 (0.03)
0.196 (0.03)
hopper-medium--e exper
0.158 (0.044)
0.621 (0.455)
0.621 (0.455)
0.203 (0.025)
0.235 (0.034)
0.203 (0.025)
walker2d-expert
0.127 (0.021)
2.607 (0.856)
0.101 (0.02)
walker2d- walker2d-medium-ooo mwdi-m-eddiumeeeeeeeeee expert
0.144 (0.03)
0.421 (0.141)
0.144 (0.03)
walker2d-medium
0.37 (0.037)
1.416 (0.667)
0.287 (0.038)
Table 2: We ablate over different mechanisms for hyperparameter selection for the learned simulator. We present the design errors. We find that OTED's use of expected log ratios for hyperparameter selection performs best.
We show our MiniGrid results in Figure $ where we present absolute error between learned pa- rameter and ground truth parameter. After be- ing trained for only several hundred steps, the discriminator learns to distinguish between gen- erated and expert data, emitting very strong signal for the designer to learn the simulator pa-
Parameter
Absolute Error
0.20
rameter. Note that 0.26 is not in the support of the parameter distribution as we quantize by increments of 0.05 but OTED is still able find the optimal prediction. This shows (i) the capa- bility of OTED to generalize to observations that are not seen during training and (ii) the robust- ness to quantization errors. This also explains why it takes slightly longer for the discriminator to learn.
Timestap
Figure 3: Absolute design error of OTED using different target design parameters MiniGrid.
6 CONCLUSION
We introduced a principled method, OTED, to learn simulator parameters of target environment from offline datasets, combining two different paradigms, namely offline and online RL. We proposed a KL- divergence regularized distribution matching objective where distribution learned to minimize the discrepancy between the offline dataset and state-action distribution of online behavior policy in the induced simulator. We also introduced unsupervised simulator and
simulator parameters

---

Under review as a conference paper at ICLR 2022
policy selection methods. By jointly learning the behavior policy and design distribution, we showed that OTED is able to recover the unknown parameter with high accuracy. When evaluated on previous benchmarks, including D4RL, OTED exhibited low absolute error average. The proposed model selection method consistently outperforms baselines and leads to successful online agent training. An online SAC agent (OTED-SAC) trained with designed simulators performed previous shateooftthe-at offfine and online RL methods. While it is still an open question what is an ideal simulator interface, believe this depends on the problem domain. For example, friction is an important parameter in robotics which depends on the type of environment such as water or air. We also would like to emphasize that any available parameter can be made explicit and learned using offline data. We showed that using some domain knowledge to set some of the parameters and reduce parameter space plausible and can give competitive results while reducing complexity. As future directions, we are looking forward to testing OTED in more complex environments such as UI navigation with real human traces.
par or better than
REFERENCES
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. optimistic perspective on offline reinforcement learning, 2020.
Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. Opal: Offline primitive discovery for accelerating offline reinforcement learning. arXiv preprint arXiv:2010.1360 13611, 2020.
Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik's bs cube with robot hand. arXiv preprint 1rXiv:1910.07111 2019.
OpenAI:
Marcin Andrychowicz, Bowen Baker, Macick Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-20, 2020.
Arthur Argenson and Gabriel Dulac-Amold. Model-based offline planning. arXiv preprint arXiv:2008.0m 05556, 2020.
Sylvain Arlot and Alain Celisse A survey of cross-validation procedures for model selection. Statistics surveys, 4:40-79, 2010.
Noam Brown and Tuomas Sandholm. Superhuman ai for multiplayer poker. 365inne,,35564455 885-890, 2019. ISSN 0036-8075, doi: 10.1 00.1166cceeeccc.ay240000 URL httpss//sciennee
Yevgen Chebotar, Ankur Handa, Viktor Makoviychuk, Miles Macklin, Jan Issac, Nathan Ratliff, and Dieter Fox. Closing the sim-to-real loop: Adapting simulation randomization with real world experience. In 2019 International Conference on Robotics and Automation (ICRA), pp. 8973-8979. IEEE, 2019.
Maxime Chevalier- Lucas Willems, and Suman Pal. Minimalistic gridworld environment for openai gym. Attps: /ettps://gituss com hwttm//maaimmebbccmmmm htteee/cb///yyymeeeee soym-minigrid 2018.
Hao-Tien Lewis Chiang, Aleksandra Faust, Marek Fiser, and Anthony Francis. Learning navigation behaviors end-to-end with autorl. IEEE Robotics and Automation Letters, 4(2):200--2014.200 2019. doi:
Damien Ernst, Guy-Bart Stan, Jorge Goncalves, and Louis Wehenkel. Clinical data based optimal sti strategies for hiv: reinforcement learning approach. In Proceedings of the 45th IEEE Conference on Decision and Control, pp. 667-672. IEEE, 2006.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint 07X2v:2004.07219, 2020.
Justin Fu, Mohammad Norouzi, Ofir Nachum, George Tucker, Ziyu Wang, Alexander Novikov, Mengjaa Yang. Michael R Zhang, Yutian Chen, Aviral Kumar, et al. Benchmarks for deep off-policy evaluation. arXiv preprint arXiv:2 16596, 2021.
10

---

Under review as a conference paper at ICLR 2022
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pp. 2052-20622 2019.
Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. arXiv preprint arXiv:1406.2661, 2014.
Caglar Gulcehre, Ziyu Wang. Alexander Novikov, Tom Le Paine, Sergio Gómez Colmenarejo, Konrad Zolma, Rishabh Agarwal, Josh Merel, Daniel Mankowitz, Cosmin Paduraru, et al. RI unplugged: Benchmarks for offline reinforcement learning. arXiv preprint WxXiv2206.138800 2020.
Ishaan Gulrajani, Faruk Ahmed, Martín Arjovsky, Vincent Dumoulin, and Aaron C. Courville. Improved training of wasserstein gans. In NIPS, 2017.
Izzeddin Gur, Natasha Jaques, Yingjie Miao, Jongwook Choi, Manoj Tiwari, Honglak Lee, and Aleksandra Faust. Environment generation for zero-shot compositional reinforcement learning. In NeurlPS, 2021.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Schoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algorithms and applications, 2019.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Gamnet (eds.), Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016. URL
neu ps. cc/ Paper.
/agpe//2016/5le//cc/eee http:///ccccc.c.ooooooooaoaaaaa.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Iharz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, and Sergey Levine. Scalable deep reinforcement learning for vision-based robotic manipulation. In Aude Billard, Anca Dragan, Jan Peters, and Jun Morimoto (eds.). Proceedings of The 2nd Conference Robot Learning, volume 87 of Proceedings of Machine Learning Research, pp. 651-673. PMLR, 29-31 Oct 2018.
Ilya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tompson. Discriminator-actor-rritt Addressing sample inefficiency and reward bias in adversarial imitation learning. In International Conference on Learning Representations, 2019. URL https:// openreview httpeeiwwwwwweeeeeeeeeeeeeeeeeee ftrt///oumm/aeemm 1d=Hk4
Ilya Kostrikov, Olir Nachum, and Jonathan Tompson. Imitation learning via off-policy distribution matching. In International Conference on Learning Representations, 2020. URL https://
Ilya Kostrikov, Jonathan Tompson, Rob Fergus, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. arXiv preprint arXiv:2103. 08050, 2021.
Srivatsan Krishnan, Behzad Boroujerdian, William Fu, Alcksandra Faust, and Vijay Janapa Reddi. Air learning: deep reinforcement learning gym for autonomous aerial robot visual navigation. Machine Learning, 1000))2501 - 2021. doi: 10. 12.10007
URL htthttss
//doi.org
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems, 2019.
Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-to- online reinforcement learning via balanced replay and pessimistic q-ensemble. arXiv preprint arXiv:2107.00591, 2021.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems, arXiv preprint arXiv:2005.0/66 2020.
Jan Matas, Stephen James, and Andrew Davison. Sim-to-real reinforcement learning for deformable object manipulation. In Conference on Robot Learning, pp. 734-743. PMLR, 2018.

---

Under review as a conference paper at ICLR 2022
Tatsuya Matsushima, Hiroki Furuta, Yutaka Matsuo, Ofir Nachum, and Shixiang Gu. Deployment- efficient reinforcement learning via model-based offlin optimization. S0HV::0006003647, 2020.
arXiv preprint
Bhairav Mehta, Manfred Diaz, Florian Golemo, Christopher Pal, and Liam Paull. Active domain randomization. In Conference on Robot Learning, pp. 1p..11666 1162-1176. PMLR, 2020.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charlie Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518:529-5333 2015.
Ofir Nachum, Michael Ahn, Hugo Ponte, Shixiang Gu, and Vikash Kumar. Multi-agent manipulation via locomotion using hierarchical sim2real. arXiv preprint arXiv: 100.:v/000.0524, 201 9a.
Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. Dualdice: Behavior-agnostie estimation of discounted stationary distribution corrections. arXiv preprint NIXX::1906.047333 2019b.
Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. Algaedice:
Policy gradient from arbitrary experience. arXiv preprint arXiv: 19/v02192,.02074, 2019c.
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. reinforcement learning with offline datasets. 2020.
Awac: Accelerating online
XuanLong Nguyen, Martin J Wainwright, and Michael Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions Information Theory, 56(11)55877-5811 2010.
Tom Le Paine, Cosmin Paduraru, Andrea Michi, Caglar Gulcehre, Konrad Zoina, Alexander Novikov, Ziyu Wang, and Nando de Freitas. Hyperparameter selection for offline reinforcement learning. arXiv preprint wwXiv:2007.09055. 2020.
Dean A Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural computation, 3(1):88-97, 1991.
Fabio Ramos, Rafael Carvalhaes Possas, and Dicter Fox. Bayessim: adaptive domain randomization via probabilistic inference for robotics simulators. arXiv preprint (9ti::190000122,, 2019.
Fereshteh Sadeghi and Sergey Levine. Cad2rl: Real single-image flight without single real image. arXiv preprini arXiv: I.02201, 2016.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889--197,, 2015.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schritwiesser, Ioannis Antonoglou, Veda Panncershelvam, Marc Lanctot, et al. Mastering the game go with deep neural networks and tree search. nature, 529(7587):484-489, 2016.
Bradly C Stadie, Pieter Abbecl, and Ilya Sutskever. third-person imitation learning. arXiv preprint 2017.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 23-30. IEEE, 2017.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEFF/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Varun Tolani, Somil Bansal, Aleksandra Faust, and Claire Tomlin. Visual navigation among humans with optimal control as a supervisor. IEEE Robotics and Automation Letters, 6(2):2288-22 2298, 2021. doi: 10.1 109/ RA 202..022.30000 3066638.
12

---

Under review as a conference paper at ICLR 2022
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint ar..v: 2019.
Tengyang Xie, Nan Jiang, Huan Wang. Caiming Xiong, and Yu Bai. Policy finetuning: Bridging sample-efficient offline and online reinforcement learning. arXiv preprint arXiv:2106.0448: 2021.
Mengjiao Yang and Ofrr Nachum. Representation matters: Offline pretraining for sequential decision making. arXiv preprint arXiv:2102.058/5 2021. 2021.
Mengjiao Yang, Bo Dai, Ofir Nachum, George Tucker, and Dale Schuurmans. Offline policy selection under uncertainty. arXiv preprint arXiv:2012.06919, 2020.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. arXiv preprint arXiv:2005.13225 2020.
13


Reviews:
[{"review_id": "YlFfa--eqOw", "reviewer": "Reviewer_2itJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1.\tStrengths and Weaknesses\n\n\\+ The idea of using an offline dataset to calibrate the simulator is intuitive and interesting. This is an alternative way of using the offline datasets to help the training of decision-making agents.\n\n\\- The proposed method is relatively heuristic and may have limitations on the novelty. As the authors mentioned, using the policy gradient method to optimize simulator parameters could have a large variance. The proposed solution (picking the best hyperparameter after multiple fittings) in this paper may not be applicable in high-dimensional cases due to the high computational cost.  \n\n\\- If I understand correctly, one assumption of the proposed method is that the model of the simulator is correct and only the simulator parameters are inaccurate. If so, this would limit the application of this method in more complex tasks, where we are unable to access or build a correct model. I suspect that the Offline RL method may outperform the proposed method.\n\n\\- There are lots of components in multiple stages in the proposed method. However, the analyses of these components are not enough. For example, the authors use a behavior cloning (BC) method as a surrogate model to collect data from the simulator. Why do they select BC methods and why is this a good choice? What if the BC model is not well-trained and makes the learning of the simulator fail?\n\n2.\tQuestions\n\n(1)\tWhat if we don’t have an accurate model of the simulator? For example, there could be some friction parameters that are not considered in simple simulators. In this case, the performance of the proposed method will always have a gap. In some worst cases, the learned simulator is too simple to fit the offline data and the online agents overfit the simulator. Could the author provide more experiments on an inaccurate simulator model with fewer parameters than the testing environment?\n\n(2)\tThe joint training of the discriminator and parameter generator could be unstable. For example, a strong discriminator may make the generator fail. How do the authors deal with this problem?\n\n(3)\tIn equation (3), a hyperparameter $\\epsilon$ is introduced and the authors say that it can balance exploitation and safety. What does safety mean here? I assume this paper is not related to safety since this term is only mentioned here.\n\n(4)\tWhat’s the influence of the seed $i$ in Section 4.2? How large is $i$ enough to reduce the high variance? Are there any theoretical or empirical analyses about it?\n", "summary_of_the_paper": "Motivated by the gap between training environments and testing environments, this paper proposes to use offline data to calibrate the simulator then use online RL methods to train generalizable agents. To minimize the divergence between offline data and simulator parameters, the authors train a discriminator. They also propose two cross-validation methods to select the best simulator parameters to increase the performance.\n\n", "summary_of_the_review": "In general, this paper proposes a very interesting direction, but important analyses and experiments are missing in the current version. I tend to reject this paper before the discussion based on my concerns in the main review. I may raise the score if the authors can provide reasonable answers to my questions.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "WYR_vb8hXQy", "reviewer": "Reviewer_rXE7", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n\nOverall, I think this paper could be a valuable contribution at ICLR 2022 since it is one of the first to make a reasonably convincing case for the effectiveness of combining offline and online RL. Moreover, it shows how one can leverage (a limited amount of) prior knowledge about the environment to create an accurate simulator, while inferring from data the parts which are unknown / uncertain. I believe is a suitable setting for many problems of interest. \n\nThe paper is clearly written, technically correct, the proposed approach is novel (as far as I know) and evaluated quite extensively on a variety of tasks, and compared with strong baselines and ablations. \n\n### Weaknesses\n\nI think the paper could use some improvements and there are some missing experiments / analysis that could make it stronger, as detailed below:\n\n1. It looks like you are missing the results on MiniGrid. Can you please include and describe them in the rebuttal? It would be useful to know how well OTED performs when learning from images in discrete domains, in order to fully evaluate the contribution and the generality of the method across different benchmarks.\n\n2. Is there a reason for which you are not comparing with adaptive domain randomization (ADR; OpenAI et al. 2019) or your DR baseline essentially ADR? Can you please include more details about the DR baseline and explain how it differs from ADR or add the comparisons with ADR?\n\n3. How does performance change with the accuracy of the learned simulator parameters? This could shed light into the robustness of the method with respect to errors in the learned simulators. This could also help better understand the trade-offs between the compute needed to learn the simulator and the performance gains. \n\n4. I think the paper should lay out more transparently the kinds of assumptions made regarding the simulator. It should also include a discussion regarding deviations from these assumptions such as large number of parameters that need to be inferred or the possibility of having a wrong model of the simulator. How could one discover the parameters that need to be inferred or detect model misspecifications? I think more context could help readers to understand when this method is suitable and when it is not and make it clear that it does use some prior knowledge about MDP. \n\n5. The paper is also missing a discussion of the limitations of the proposed approach and future directions stemming from these ideas. These are important to place the work in the larger research context and inspire follow-up contributions. \n\n6. Another concern I have is regarding how the performance (and simulator accuracy) scale with the number (or range) of the parameters you need to learn. For example, if you have less prior knowledge about the simulator, you may need to infer multiple parameters. I think an experiment showing how this scales would provide additional insight. \n\n7. The related work section seems to be missing some relevant work such as \"AWAC: Accelerating Online Reinforcement\nLearning with Offline Datasets\" by Nair et al. 2021. This paper also uses a combination of online and offline RL (although it assumes limited access to the true simulator), so a discussion of the two is needed. \n\n### Minor \n\n1. Do you have an intuition regarding why there is divergence in Figure 2 for gravity in walker and for all params in hopper?\n2. Can you bold the model with the best performance in the tables for readability?\n3. Please proof read. There are a few typos on pages 1, 6, 8, and likely more. \n \n\n", "summary_of_the_paper": "This paper proposes a new approach, which combines offline reinforcement learning with learning in simulation, without the need of training on the target environment. More specifically, they propose to use offline data to learn a distribution over the simulator's parameters and then use the inferred simulator to train an RL policy online. The experiments show that the proposed approach, OTED, performs significantly better than other offline RL, behavior cloning, and domain randomization methods, particularly when learning from non-expert demonstrations as well as in the low-data regime. ", "summary_of_the_review": "Given all the above, I am currently leaning towards recommending acceptance. If the authors address the above concerns (adding MiniGrid experiments, ADR comparisons, analysis of how the performance scales with the simulator's accuracy, and a discussion of the method's limitations), I plan to recommend acceptance (unless the other reviewers change my mind or realize I have missed some crucial details). ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "PYI0MSErLCu", "reviewer": "Reviewer_huR6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strengths\n\n1) Novel approach\n\t- Offline RL methods have generally focused exclusively on the policy learning process. Shifting focus to the environment parameterization is a valuable change in approach. Previous efforts in this vein have been limited to efforts like domain randomization that do not attempt to learn a good simulator for the target task.\n2) Promising simulator fitting results\n\t- Results are promising for the degree of accuracy in the simulator fits. There is clearly room to improve when allowing fits of many parameters, but the initial results are encouraging.\n\n\n# Weaknesses\n\n1) Modest policy performance\n\t- Given the amount of tuning of the simulator and policy selection processes, the results in Table 1 show very modest results. Behavior cloning seems to generally be a viable alternative to the simulator in most cases and is substantially easier to implement and train.\n\t\t- Action: Explain why BC would offer such strong performance.\n\t\t- Action: Perhaps add an explanation of the additional benefit to having a simulator vs only the final policy.\n\t- Table 1 does not seem to fully support the claim \"We see almost no performance drop when all parameters are trained jointly\": both hopper-medium-expert (~20 points lower) and hopper-expert (30 points lower than the next worst) values seem much lower than the single parameter variants.\n\t\t- Action: Could this be explained in terms of the environment dynamics? (Or connected to the results in Appendix D with difficulty of learning the hopper environment?)\n2) Lack of error bars / statistical tests for policy training results\n\t- The core results on policy training (Table 1) have no measure of variance or a statistical test of the differences among methods.\n\t- I recognize the method is computationally intensive to train, but these results are weakened by the lack of any statistical validation of the outcomes. Repeated trials are necessary to establish confidence in the outcomes.\n\t- I highlight this outcome as it is the core end goal of using simulator tuning: obtaining a better policy for the real task.\n\t- Action: Add some form of repeated trials and statistical differences test to the policy training results.\n3) Lack of statistical tests of differences for the simulator ablations\n\t- Action: Add repeated trials and statistical tests to show differences (or lack thereof) among the ablations of OTED.\n\nNote: The abstract mentions strong performance with as few as 5 demonstrations. I could not find where these results were reported.\n\nMinor note: I would suggest choosing a different name for the parameter $\\epsilon$ as this is typically the symbol used to indicate a noise parameter for sampling distributions (or a very small scalar quantity). For example, $\\lambda$ is a widely used name for a trade-off weighting parameter.\n", "summary_of_the_paper": "The paper introduces an approach to tune a parameterized simulator to produce data samples that mimic a given offline dataset of trajectories from real data. The core technical contributions involve defining a KL divergence to quantify the difference of the simulator data to a given offline dataset, a training algorithm that interleaves simulator tuning and agent training on simulator data, and selection algorithms to choose a hyperparameterization among simulators and a policy among a set trained on a simulator. Evaluations assess how well the training procedure can recover known simulator parameters and the performance of policies trained on a learned simulator to policies trained with offline RL algorithms on the offline data directly. Results show the simulator can accurately recover parameters and is able to obtain similar performance to offline RL algorithms when tuning certain parameters.", "summary_of_the_review": "The paper presents an important approach to address offline RL training by tuning a parameterized simulator to produce training data for a policy. This has great potential for further investigation and comparison/combination with other offline RL approaches. The empirical results are weak in the current form, leaving me less certain the work is mature enough for publication. I incline toward seeing this as sufficient contribution to merit acceptance.\n\nMy score would improve if the authors can provide clear statistical evidence of the superiority of the new method.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "LUpX_YCAh0t", "reviewer": "Reviewer_mcpN", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n- The proposed method for learning simulator parameters seems well motivated and supported by compelling empirical evaluations. \n- Performance comparisons seem thorough and the method seems especially helpful with lower quality demonstrations, since it only uses the suboptimal demonstrations for simulator parametrization and not policy learning.  Overall, the idea of using an offline dataset of suboptimal demonstrations to learn simulator parameters for online learning is interesting. \n\n### Weaknesses\n- The abstract mentions using this method to close the sim2real gap, however it is hard to say how well this will extrapolate to sim2real applications, as this relies on the offline dataset and online learning domains to be identically parametrized. i.e. even if we have an offline dataset of real world demonstrations, if our simulator can’t isolate and identify the correct parameters to adapt (which is likely much harder with the sim2real domain gap compared to the simulated gap here), would the method still work? \n- The design deviation method also seems comparable to OTED with respect to design errors (in Table 2) — are there evaluations on task performance with this scheme? \n- Missing standard deviations to results presented in Tables 1, 2.  \n\n### Questions \n- How sensitive is the parameter learning to the approximate behavior policy learned (i.e. is a reasonably good policy necessary or will a random policy be sufficient for learning simulator parameters)? \n\n### Additional Feedback\n- Would be interesting to include an ‘oracle’ method — online SAC with ground truth parameters to see the gap with OTED-SAC, if any. ", "summary_of_the_paper": "The authors propose a method, Offline Targeted Environment Design, OTED, for using offline datasets to train ‘realistic’ simulators by minimizing the divergence between the state-action distributions of the offline dataset and in the learned simulator. The authors also introduce methods for selecting learned simulators and policies across simulators, and compare their method against a range of offline RL methods and online RL methods with variations of domain randomization. ", "summary_of_the_review": "The method is intuitive and it seems like it is capable of learning the chosen simulator parameters well, potentially leading to novel methods in combining system identification and offline RL. However, it is unclear how necessary the more complex sampling method is (compared to design deviation), and applicability to real world system identification problems seems limited. The paper would be strengthened by examining how well the method works in adapting 'realistic' simulators from real world offline data, where there is more likely to be interactions between multiple simulator parameters and the domain gap is larger. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new approach, which combines offline reinforcement learning with learning in simulation. There were different views on the paper among the reviewers and we had quite a lot of discussions. As a consequence, there were still serious concerns remaining, e.g., whether the results are significant enough, whether there are clear advantages of the proposed mehtod over directly using offline RL methods. It is not justified whether the proposed framework can use offline data more efficiently or better reduce the gap between mismatched simulators and offline data. The reviewer who gave the highest score decided not to champion the paper. Considering all the discussions, we believe the paper is not ready for publication at ICIR yet.
Decision: Reject