Marginalized Importance Sampling for Off-Environment Policy Evaluation
Authors: Pulkit Katdare
Nan Jiang
Katherine Rose Driggs-Campbell
Venue: CoRL 2023 Poster (2023)
Keywords: Sim2Real
Policy Evaluation
Robot Validation

Abstract: Reinforcement Learning (RL) methods are typically sample-inefficient, making it challenging to train and deploy RL-policies in real world robots. Even a robust policy trained in simulation requires a real-world deployment to assess their performance. This paper proposes a new approach to evaluate the real-world performance of agent policies prior to deploying them in the real world. Our approach incorporates a simulator along with real-world offline data to evaluate the performance of any policy using the framework of Marginalized Importance Sampling (MIS). Existing MIS methods face two challenges: (1) large density ratios that deviate from a reasonable range and (2) indirect supervision, where the ratio needs to be inferred indirectly, thus exacerbating estimation error. Our approach addresses these challenges by introducing the target policy's occupancy in the simulator as an intermediate variable and learning the density ratio as the product of two terms that can be learned separately. The first term is learned with direct supervision and the second term has a small magnitude, thus making it computationally efficient. We analyze the sample complexity as well as error propagation of our two step-procedure. Furthermore, we empirically evaluate our approach on Sim2Sim environments such as Cartpole, Reacher, and Half-Cheetah. Our results show that our method generalizes well across a variety of Sim2Sim gap, target policies and offline data collection policies. We also demonstrate the performance of our algorithm on a Sim2Real task of validating the performance of a 7 DoF robotic arm using offline data along with the Gazebo simulator.

Marginalized Importance Sampling for Off-Environment Policy Evaluation
Pulkit Katdare
Department of Electrical and Computer Engineering University of Illinois Urbana- Champaign Illinois, United States katdare20illinois edu
Nan Jiang
Department of Computer Science
University of Illinois Jrbana-Champaiin Illinois, United States
Katherine Driggs-Campbell
Department of Electrical and Computer Engineering University of Illinois Urbana- Urbana-Champalla Champaign Illinois, United States
Abstract: Reinforcement Learning (RL) methods are typically sample-inefficien making challenging to train and deploy RL-policies in real world robots. Even a robust policy trained in simulation requires a real-world deployment to assess their performance. This paper proposes a new approach to evaluate the real-world performance of agent policies prior to deploying them in the real world. Our approach incorporates simulator along with real-world offline data to evaluate the performance of any policy using the framework of Marginalized Importance Sampling (MIS). Existing MIS methods face two challenges: (1) large density ratios that deviate from a reasonable range and (2) indirect supervision, where the ratio needs to be inferred indirectly, thus exacerbating estimation error. Our approach addresses these challenges by introducing the target policy's occupancy in the simulator as an intermediate variable and learning the density ratio the product of terms that can be learned separately. The first term is learned with direct supervision and the second term has small magnitude, thus making it computationally efficient. We analyze the sample complexity as well as error propagation of our two step-procedure. Furthermore, we empirically evaluate our approach on Sim2Sim environments such as Cartpole, Reacher, and Half-Cheetah. Our results show that our method generalizes well across a variety Sim2Sim gap, target policies and offline data collection policies. We also demonstrate the performance of our algorithm Sim2Real task of validating the performance of DoF robotic arm using offline data along with the Gazebo simulator.
Keywords: Sim2Real, Policy Evaluation, Robot Validation
Introduction
Reinforcement Learning (RI.) algorithms aim to select actions that maximize the cumulative returns over finite time horizon. recent years, RL has shown state-of-thean performance over range of complex tasks such as chatbots, [1], games [2], and robotics [3, 4. 5, 6]. However, RL algorithms still require large number of samples, which can limit their practical use in robotics [7, 8).
typical approach is to train robust robot policies in simulation and then deploy them on the robot [9, 10). Such an approach, although useful, does not guarantee optimal performance on real robot without significant fine tuning [11]. this work, we propose an approach that evaluates
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

the real world performance of policy using robot simulator and offline data collected from the real-world [6]. To achieve this, we employ the framework of off-policy evaluation (OPE) [12].
OPE is the problem of using offline data collected from a possibly unknown behavior policy to estimate the performance of different target policy. Classical OPE methods are based on the principle of importance sampling (IS) [13, 14, 15], which reweights each trajectory by its density ratio under the target versus the behavior policies. More recently, significant progress has been made on marginalized importance sampling (MIS), which reweights transition tuples using the density ratio (or importance weights) over states instead of trajectories to overcome the so-called curse of horizon [16, 17, 18]. The density ratio is often learned via optimizing minimax loss functions.
Most existing MIS methods are model-freee relying data from the real environment to approx- imate the MIS weight function. However, direct application of MIS methods to robotics carries two main disadvantages. (I) Large ratios: MIS method learns distribution mismatch between the behavior and the target policies. When the mismatch between the behavior and the target policy is large, which is often the case, MIS method tend be challenging to learn. (2) Indirect supervision: MIS methods requires no samples from target policies, which requires the weight being learned in- directly via the Bellman flow equation. states where coverage of the offline dataset is scarce, MIS methods tend to perform poorly.
In robotics, it is reasonable to assume access to good but imperfect simulator of the real environ- ment [19, 20, 21J. In this work, we propose an improved MIS estimator that estimates the density ratio mismatch between the real world and the simulator. We further show that such a MIS estimator can be used evaluate the real-worke performance of a robot using just the simulator. As described in figure 1, we estimate the discrepancy between the real world and the simulator by using the target policy's 's occupancy in the simulator an intermediate variable. This allows to calculate the MIS weights combination of two factors, which can be learned independently. The first factor has di- rect supervision, while the second factor has small magnitude (close to 1), thereby addressing both large ratios and indirect supervision issues mentioned above. We present a straightforward deriva- tion of our method, examine its theoretical properties, and compare it to existing methods (including existing ways of incorporating a simulator in OPE) and baselines through empirical analysis.
(1) We derive MIS estimator for off envirronneenn evalu- ation (Section 4). (2) We explore the theoretical properties of our off-environment evaluation es- timator by proposing a sample-complexity analysis (Section 4) and studying its special cases in linear and kernel settings. (3) We empirically evaluate our estimator on both Sim2Sim as well as Sim2Real environments (Section 5). For the Sim2Sim experiments, we perform a thorough ablation study over different Sim2Sim gap, data-collection policies and target policies, environments (Taxi, Cartpole, Reacher, and Half-Cheetah). Furthermore, we demonstrate practicality of our approach on sim2real task by validating performance of Kinova robotic arm using offline data along with Gazebo based Kinova simulator.
We make the following contributions.
2 Preliminaries
Robot learning problems are often modelled as an infinite-horizon discounted Markov Decision Process (MDP). MDP is specified by (S,A, P. R. 7, do). Here, S and A are the state and the action spaces, P : S x A + A(S) is the transition function (A(-) is the probability simplex). We also define the reward function R : S x A + A[O,, Rmaz]) 1), E [0, 1) is the discount factor, and dp E A(S) is the initial state distribution. A policy π : S + A(A) induces a distribution of trajectory: Vt 2 0, 8o ~ do. ag ~ x(-1s)).re re ~ R(-ss,),, 81+1 ~ P(-]s,, at). The performance of π measured by its expected discounted return under the initial state distribution, defined as JP(T) (1 - >> 7're/n |,, doj; here, we use the subscript P in JP(z) to emphasize the dynamics w.r.t. which the return is defined, since will consider both the true environment and the simulator in the rest of this paper and the subscript will help distinguish between them. (TT)) also

---

On-Policy Simalator
B
Egg [w.r]
Off-invionement
Computes
Computes W
Evaluation Score
db
Figure 1: For given policy πT, we first collect on-policy data di,,, from simulator environment. Using d and offline data d& on the real world, first calculate the importance sampling fac- tor 3 = - This importance sampling factor essentially allows re-weight existing off-policy evaluation algorithm to estimate /d%, which helps us estimate real-world per- formance of the agent JP.C(T) using on-policy simulator data.
has an alternative expression (T) =- (. [r]. where
dp(s,a) 71 Ei= 7'p(s; s,a, al
is the discounted state-action occupancy induced by from dg- An important quantity asso- ciated with policy is its Q-function Q7, which satisfies the Bellman equation (7,(s, a) Ry [r + (Qp(s' (s', rT), where f(s', T) == Eg Ea'=a( a')]. We can also define the state-value function V (s) = (7(s, and J(x) OPE and Marginalized Importance Sampling: In off-policy evaluation (OPE), we want to eval- uate a target policy π using data collected from a different policy in the real environment, de- noted by its dynamics (s,a) ~ prr R(s,a),& ~ P(s, a), and the sample size is n. When the data is generated from some behavior policy Th. μ can correspond to its occupancy dp. We will use [. as shorthand for taking expectation over (s, a, s') generated from such data distribution in the real environment.
B) [V)(s)] (V)
As standard simplification, we assume data is generated i.i.d. as
The key idea in marginalized importance sampling (MIS) is to learn the weight function wp (s, = (7.(a.)
pla,a) Once this function is known, ((T) can be estimated as J(T)
r). Note that E,,[-] can be empirically approximated by the dataset sampled from the real environment. The real challenge in MIS how learn w 7// Ex- isting works often do so by optimizing minimax loss functions using Q-functions as discriminators, and is subject both difficulties (large ratios and indirect supervision) mentioned in the introduc- tion. We refer the readers to [22] for summary of typical MIS methods.
((s.)) [r] E/
Learning Density Ratios from Direct Supervision: Given two distributions and over the same space X, the density ratio p(z)/q(<) can be learned directly if we have access to samples from both p and g, using the method proposed by [23]:
p(x)
arg max Ez-p [(x) Bz~g[((x)] + J:X
(x))
To guarantee generalization over finite sample when we approximate the expectations empirically, we will need to restrict the space f that we search over to function classes of limited capacities, such as RKHS or neural nets, and p(x)/q (x(x) can still be well approximated as long as it can be represented the chosen function class (i.e., realizable). More concretely, we have n samples TI..... Zn from p and m samples .i....,ā,, from q, and use F approximate p(z)/q(x). the learned density ratio be made generalizable by adding regularization term T (J) to improve the statistical and computational stability of learning:
E(1z)) A 110)22
arg max JEF n
Σ f(x;)
(3)
3

---

3 Related Work
3.1 Reinforcement Learning applications in Robotics
There are three different themes that arise in reinforcement learning for robotics [24, 25]. (1) Sim2Real: algorithms are primarily concerned with learning robust policy in simulation by train- ing the algorithms over a variety of simulation configurations [26, 9, 27]. Sim2Real algorithms, although successful, still require a thorough real-world deployment in order to gauge the policy's performance. (2) Imitation learning algorithms learn an optimal policy by trying to mimic offline expert demonstrations [28, 29, 30]. Many successful imitation learning algorithms minimize some form of density matching between the expert demonstrations and on-policy data learn optimal policy. A key problem with imitation learning is the fact that it requires constant interaction with the real-world environment in order to learn an optimal policy. (3) Offline reinforcement learn- ing is a relatively new area. Here the idea is to learn an optimal policy using offine data without any interaction with the environment [31, 32, 33, 34J. Offline reinforcement learning has recently demonstrated performance at-par with classical reinforcement learning a few tasks. However, of- fline reinforcement learning algorithms tend to overfit on the offline data. Thus, even offline learning modules too require actual deployment in-order to assess performance.
3.2 Off-Policy Evaluation
We review related works in this section, focusing comparing to existing OPE methods that can leverage the side information provided by an imperfect simulator. (1) Marginalized Importance Sampling (MIS): MIS methods tend to assume the framework of data collection and policy eval- uation being on the same environment. To that end, there are both model-free [35, 36, 37) and model-based variants of MIS methods and face the aforementioned two challenges (large magnitude of weights and indirection supervision) simultaneously. Model based variants of MIS sometimes tend to be doubly robust (DR) in nature [18, 38] and benefit from Q-functions as control vari- ates, which can be supplied by the simulator. However, the DR version of MIS is meta estimator, and the weight / still needs to be estimated via a "base" MIS procedure. Therefore, the in- corporation of the simulator information does not directly address the challenges we are concerned with, and there is also opportunity to further combine our estimator into the DR form of MIS. (2) Model-based methods: Model-based estimators first approximate the transition dynamics of the target environment [39], which is further used to generate rollouts and evaluate performance for any target policy. One way of incorporating given imperfect simulator in this approach to use the simulator as "base predictions," and only learn an additive correction term, often known as residual dynamics [40]. This approach combines the two sources of information (simulator and data from target environment) in very different way compared to ours, and are more vulnerable to misspeci- fication errors than model-free methods.
4 Weight Estimator
Recall that our goal is to incorporate given simulator into MIS. We will assume that the simulator shares the same A,A, 7, dg with the real environment, but has its own transition function Per which can be different from Pte. As we will see, extension to the case where the reward function is un- known and must be inferred from sample rewards in the data is straightforward. and for simplicity we will only consider difference in dynamics for most of the paper.
Split the weight: The key idea in approach is split the weight T/" into two parts by intro- ducing di as an intermediate variable:
dP., (s,
(s,a)
dp. (s,
x/P wp (s. a)
((s,a)
(()
(8
:=9 (direct supervision)
Pie /P, (magnitude ~ 1)
Note that d'. is the occupancy of in the simulator, which we have free access to. The advantage of our approach is that by estimating 3 and w wp./PPr separately, we avoid the situation of running

---

into the two challenges mentioned before simultaneously, and instead address one at each time: B = dy.., /p has large magnitude but can be learned directly via [23] without the difficult minimax optimization typically required by MIS, and we expect w Pc- = d.p.. /ddp,_/d. to be close to 1 when Pte R Ptr (and thus easier to learn).
Estimate w P.. Since 3 is handled by the method of [23], the key remaining challenge is how to estimate P.,/P,.. (Interestingly, 3 also plays a key role in estimating w P,,/ will be shown below.) Note that once we have approximated /P.,/P... with some w, we can directly reweight the state-action pairs from the simulator (i.e., d'p,, i the reward function is known (this is only assumed for the purpose of derivation), i.c.,
Jrio (TT) = (3 ((Raa)) -r),
and this becomes an identity if w - w Pr. Following the derivation in [18, 22], we now reason about the error of the above estimator for an arbitrary to derive an upper bound as our loss for learning
((saa))dB M( -p,
= (B(aa)ad x (P ww 107, (s, -
(s', x)6]
(1 B, 1Q7 [s,
1 sup Ea xP., (g(s,a) - rg(s', +])] (1 B) (9(5,
(4)
gEQ
Here di. x Pte is shorthand for (s, a) ~ pp.,, s' a(s,a). In the last step, we handle the unknown Q7 by relaxation similar to [16, 18, 22], which takes an upper bound of the error over E Q for some function class Q c SxA, and the inequality holds as long as Q7, E conv( Q) with conv(.) being the convex hull.
Approximate d',,, x Pte: The remaining difficulty that we will need samples from di x Pies i... (s, a) sampled from r's occupancy in the Ptr simulator, and the next s' generated in the Pte real environment. While there is natural dataset for such distribution, real environment, (s, a, s') px Pte. and reweight it using d / to approximate expectation w.r.t. d'p,, x Pte, i.e., (s, dy. Pie = as, P x Pte reweighted with B d.p./M. Based on such observation, we can further upper-hound [E)s,)))d , from end of Equation with:
take the data from the
M R~ r] Jr. (x)1
sup L (w, B, q) = [a[l B (g(s, a) - rg(s', x))] - (1 Bond, T)) qEQ
(5)
As our derivation has shown, this is valid upper bound of the error long conv Q) can represent Q We also need to show that the upper bound is non- non-trivial, i.e., when = WT. the upper bound should be O. This is actually easy to see, as for any
L( L( P../PP. P.. 3, g) = Edg, x Pc. [w Pie (a(s,a) (q(s',m)) (1 - lddd (6)
Eug, xx(p(saass. a) 79(s', ]] - (1 - 7) (a(s, x 0.
The last step directly follows from the fact that d',, is valid discounted occupancy and obeys the Bellman flow equation. Therefore, it makes sense search for over function class W C SxA to minimize the loss sup,e L(w, g...
Final estimator: To summarize our estimation procedure, we will first use [23] to estimate P3 dp./n with a function class F, and plug the solution into our loss for estimating w PP./P,.. i.e.
i
arg min sup (w: B.9)-
(8)
wEW D3Ω
As mentioned above, if the reward function is known, can ((s.a)-dp_.. .r-)) R(-,(s r] as our estimation of Jr.. (T). We can also demonstrate interesting properties of our optimization like the effect of linear function class and RKHS function class. This discussion is deferred to the supplementary materials section 9.1.
Sample Complexity Guarantee: We can further provide an upper-bound on the performance of our final estimator under the following two assumptions.
5

---

Assumption (Boundedness). We assume vf F, 0 < CF f S CF.max- Define Cy == F,,aax + max(log F,max= log CF min) Similarly, Vi E W, w E [w, and bq E Q q E [0,Co) Col.
Assumption (Realizability of F). Sp./EEFF
Theorem 4.1. Let B be our approximation of d'p/ which we found using [23]. We utilize this to further optimize for Wn (equation 8) using n samples. In both cases, (ss.a))d%, H is also approximated with samples from the simulator Ptr. Then, under Assumptions I and 2 along with the additional assumption that Qp,,. E (C(2) with probability at least 1 ō, the Off Environment Evaluation error can be bounded as
E (s.a)dd3 R(s,)) [tén r] JPco S min max L (w, B, gE
d7
2 (3)
+ 2OW Co
(ER...) + CT/
1
10
n
log(%)
+ (W, Q) CeCwy
2n
where (,, R. (W, Q) are the Radamacher complexities of function classes ((x, + f(x) log(f(y)) F} and a(s,a. + (w(s, Spr(s.0)
p(s.a) (g(s,a) 7g(s', π)) W,g 2}, respectively, / = maxs,a d'p (8, a)/ a)/u(s,,) a)) measures distribution shift between d'p and μ4, and Ó-) is the big-Oh notation suppressing
factiorsi
Note that we do not make realizability assumption for W in the theorem above. Realizability as- sumption is reflected in the wn sup,e [[W, 33, 4)), which equals when / /dp, E W. The remaining terms vanishes at an O /n) rate when n - 3c.
Generalizing Off-Environment Policy Evaluation: A key advantage of two-step approach is that we improve many existing off off-policy evaluation algorithm with similar two-step process. is an empirically state-of-
In this work, we use our two-step procedure with GradienDDICF--winn the-art estimator in the DICE family-can also be similarly adapted as below, which we use in our experiments. Detailed derivation for the same can be found in the supplementary materials 9.3.
5 Experiments
5.1 Sim2Sim Validation of 3-DICE
Experimental Setting: In the Sim2Sim experiments, we aim to show the effectiveness of our ap- proach across different target policies, offline dataset as well as changing sim2sim gap. We further show the effectiveness of our approach over different types of Sim2Sim enviromments like Taxi (Tabular), Cartpole (discrete-control), Reacher (continuous control), and HalfCheetah (continuous control) environments. For each of these environments, we refer the default configurations of
Table 1: Log mean squared error between the performance predicted by our 3-DICE algorithm and the real world performance of the robot. We observe that our method is able to outperform DICE based baselines by a comfortable margin.
Logio Mean Squared Error (L)
Algorithm
Kinova (Sim2Real)
Taxi
Cartpole
Reacher
Half-Cheetah
Simulator
-3,96
-0.19
-2.58
-1.09
1.18
3-DICE (Ours)
-4.38
-1.60
-4.19
-4.08
-3,42
GenDICE
-3.48
-0.13
-2.84
-2.61
-2.96
GradientDICE
-3.49
-0.59
-1.45
-3.17
-2.16
DualDICE
-3.48
-0.48
-0.99
-2.88
-2.12

---

Simulstor
GenDICE
GracientDICE
DUeIDICE
Off Environment Evaluation for
0.2
[a 0.2, 6.6.)
Logro (Mean Squared Error)
-.0
Logio (Mean Squared
-1.5
-15
-24
-25
-15
-10
15
0.6
0.7
5 0.8
Sim2Real Gap (Gravity)
(a) Target Policy = 0.2
Environment Evaluation
Logio (Mean Squared Error)
Looio (Mean Squared
0.5
Sim2Real Gep (Length)
(b) Target Policy a
Figure 2: We demonstrate the effectiveness of 3-DICE Cartpole (a) and Reacher (b) Sim2Sim environment. For Cartpole environment we demonstrate the performance of 3-DICE over for a Sim2Sim pair of {10, m5/2 Similarly for Reacher the Sim2Sim pair is (0.1m, 0.075m) for the length of the link. On left hand side, we demonstrate the effect of B-DICE with different data collection policies while keeping the target policies fixed. On the right hand side, demonstrate the impact of 3-DICE with increasing Sim2Sim gap keeping the offline data collection policy the same. We observe that our 3-DICE algorithm comfortably outperforms closest DICE based baselines.
these environments as the simulator environment. We further create a "real" world environment by changing key configurations from each of these environments. For example, we modify the transi- tion probability in taxi, gravity in cartpole, and link lengths in reacher. These kinds of configuration changes help assess the performance limits of our algorithm across variety of sim2sim gap. In table 2, we list all the different sim2sim environments configurations over which we experimented our algorithm. Typically these configurations are such that the real-world performance predicted by the simulator alone is off by 9-45%
We first collect our offline data by using a noisy pre-trained policy which parameterised by 6. Higher the 8, noisier the data-collection policy. For Taxi and Cartpole environment, we using uniform random policy for the noise, while we choose zero-mean gaussian policy for continuous environments like reacher and halfcheetah. Using this offline data well evaluate the performance of any target policy, which we parameterise by Target policy is further defined by mixture of another pre-trained policy with noise. More the a. more the randomness in the policy. Detailed experimental details along with the setup has been detailed in Appendix 9.11 Results and Observations: We present the detailed results for all the four environments fig- ures (Taxi), figures 2a, (Cartpole), figures 2b and (Reacher) and figure (HalfCheetah). For the boxplot, we fix target policy (a) and demonstrate the evaluation error for our algorithm across range of offline dataset (8) while keeping Sim2Sim gap fixed. For the line plots, demon- strate the effectiveness of our algorithm across changing Sim2Sim gap, while keeping the offline
simulator, now

---

data (6) and target policy a fixed. We also compare our algorithm against DICE baselines Gra- dientDICE [36], GenDICE [37], DualDICE [35]. DICE baselines are currently the state-of-the-art algorithm in off-policy evaluation and are known to outperform even hybrid off-policy evaluation al- gorithms. In figure 5, we also compare our algorithm against hybrid off-policy evaluation baselines for the cartpole environment (further details in section 9.11). We observe that our method is able to comfortably outperform closest DICE baselines with the help of extra simulator. These results not only empirically validate the effectiveness of our algorithm, but also point out that we can learn important information from imperfect simulated environments to help in improving RI, policies. We also observe that as the Sim2Sim gap increases the performance of our algorithm tends to decline. This means that with increasing Sim2Sim gap the amount of relevant information that can be learned from the simulator diminishes. We observe that this decline actually becomes significant when the Sim2Sim gap breaches the 60% threshold.
5.2 Real-world performance validation on Kinova Robotic Arm
We demonstrate the effectiveness of our 3-DICE algorithm for sim2real validation task on Kinova robotic arm. We first collect offline data by asking users to move the arm from one-position to another via RC controllers. Our data collection ensures sufficient coverage of the robotic arm's 's task space. We then use this offline data along with our in-house gazebo based simulator to experimentally validate the real-world performance of a PID controller using B-DICE that moves our robot from given initial location to any desired location.
Experimental Setting:
Results and Observations: Our results along with different baselines are averaged over 10 different locations are tabulated in Table 1. We observe that 3-DICE is able outperform satt--o-ttheane showing an improvement of 60% over the nearest baseline. There are two key conclusions from all of our experiments. One, although B-DICE outperforms state-of-the art baselines in off -oolcy evaluation. We observe that the performance drops when the gap between the target policy and behavior policy increases. Two, prediction error decreases as the gap between the training and test environments increases, as the transferable information between the two environments decreases.
Limitations
We present the limitations of our work that we wish to address in future work. (1) Our algorithm expects high quality data with sufficient coverage of the state-action space. Identifying the confi- dence interval of our estimator will not only ensure sample efficient evaluation, but also help us in designing robust offline reinforcement learning algorithm. (2) Similar DICE class of min-max op- timization, algorithm also suffers from high variance in their performance. Efforts required to reduce this variance.
Conclusion and Future Work
We derive a novel MIS estimator that is able to evaluate real world performance of a robot using offline data and an imperfect robot simulator. We then develop sample complexity bounds, and empirically validate our approach on diverse Sim2Sim environments and Sim2Real environment like KinovaGen3 robot. For future work, we wish to utilize this framework of off-ennvronment evaluation to learn optimal robot policies using simulation and limited amount of real-world offline data.
8 Acknowledgements
The authors thank Neeloy Chakroborty and Shuijing Liu for their valuable suggestions this paper draft. This work was supported in part by ZJU-UIÜC Joint Research Center Project No. DREMES 202003, funded by Zhejiang University. Additionally, Nan Jiang would also like to acknowledge funding support from NSF 11-21 and NSF CAREER I1S-2141711.

---

References
[1] C. K. John Schulman, Barret Zoph et al. Chatgpt: Optimizing language models for dialogue, 2023.
[2] D. Silver, Schrittwieser, Simonyan, and thers. Mastering the game of go without human knowledge. Nat., 2017.
[3] S. Liu, P. Chang, Z. Huang, N. Chakraborty, W. Liang, J. Geng, and K. R. Campblla Socially aware robot crowd navigation with interaction graphs and human trajectory prediction. 2022.
[4] OpenAI, I. Akkaya, M. Andrychowicz, et al. Solving rubik's s cube with robot hand. 2019.
[5] W. Yu, V. C. V. Kumar, G. Turk, and C. K. Liu. Sim-to-real transfer for biped locomotion. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS, 2019.
[6] P. Katdare, S. Liu, and K. D. Campbell. Off environment evaluation using convex risk mini- mization. In International Conference on Robotics and Automation, ICRA, 2022.
[7] Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In Proceedings of the 35th International Conference Machine Learning, ICML, 2018.
[8] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. 2017.
[9] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS, 2017.
[10] Y. Du, O. Watkins, Darrell, Abbeel, and D. Pathak. Auto-tuned sim-to-real transfer. In IEEE International Conference on Robotics and Automation, ICRA, 2021.
[11] G. Leibovich, G. Jacob, S. Endrawis, G. Novik, and A. Tamar. Validate on sim, detect on real model selection for domain randomization. In International Conference on Robotics and Automation, (ICRA), 2022.
[12] C. Voloshin, H. M. Le, N. Jiang, and Y. Yue. Empirical study of off-policg policy evaluation for reinforcement learning. arXiv preprint arXiv:1911.06833 2019.
[13] D. Precup, S. Sutton, and S. Singh. Eligibility traces for off-policy policy evaluation. In Proceedings of the Seventeenth International Conference on Machine Learning, 2000.
[14] N. Jiang and L. Li. Doubly Robust Off-policy Value Evaluation for Reinforcement Learning. In Proceedings of the 33rd International Conference on Machine Learning, volume 48, pages 652-661, 2016.
[15] S. Levine, A. Kumar, G. Tucker, and J. Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643. 2020.
[16] Q. Liu, I.. Li, Tang, and Zhou. Breaking the curse of horizon: Infinite-horizon off-policy estimation. In Advances in Neural Information Processing Systems, 2018.
[17] O. Nachum and B. Dai. Reinforcement learning via renchel- duality. arXiv preprint arXiv:2001.0188 2020.
[18] M. Uchara, J. Huang, and N. Jiang. Minimax Weight and Q-Function Learning for Off-Policy Evaluation. In Proceedings of the 37th International Conference on Machine Learning (ICML), 2020.

---

[19] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. OpenAl gym. arXiv preprint
2016.
[20] E. Coumans and Y. Bai. Pybullet, python module for physics simulation for games, robotics and machine learning.
org. 2016-2021.
[21] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, 2012.
[22] N. Jiang and Huang. Minimax value interval for off ooolccy evaluation and policy optimiza- tion. Advances in Neural Information Processing Systems, 33, 2020.
[23] X. Nguyen, M. Wainwright, and M. I. Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. CoRR, abs/0809.0853, 2008. URL http:// arxiv. org/ wwwpi..orgabbb//00000 /080 0000.0853.
[24] H. Sikchi, A. Zhang, and S. Niekum. Imitation from arbitrary experience: A dual unification of reinforcement and imitation learning methods. arXiv preprint arXiv:2302.08560, 2023.
[25] O. Nachum and B. Dai. Reinforcement learning via fenchel-rockafellar duality. CoRR, 2020.
[26] S. Höfer, K. E. Bekris, el. al. Sim2real in robotics and automation: Applications and challenges. IEEE Trans Autom. Sci. Eng., 2021.
[27] Q. Vuong, S. Vikram, H. Su, S. Gao, and H. I. Christensen. How to pick the domain random- ization parameters for sim-to-real transfer of reinforcement learning policies? 2019.
[28] S. Ross, G. J. Gordon, and D. Bagnell. A reduction of imitation learning and structured predic- tion to no-regret online learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, AISTATS, JMLR Proceedings, 2011.
[29] J. Ho and S. Ermon. Generative adversarial imitation learning. In Advances in Neural Infor- mation Processing Systems 29, 2016.
[30] Y. Wu, N. Charoenphakdee, H. Bao, V. Tangkaratt, and M. Sugiyama. Imitation learning from imperfect demonstration. CORR, 2019.
[31] A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforce- ment learning, arXiv preprint arXiv:2006.044 044799 2020.
[32] Lee, W. Jeon, Lee, J. Pineau, and Kim. Optidice: Offline policy optimization via stationary distribution correction estimation. 2021.
[33] T. Yu, A. Kumar, R. Rafailov, A. Rajeswaran, S. Levine, and C. Finn. COMBO: conserva- tive offline basel-based policy optimization. In Advances in Neural Information Processing Systems 34, 2021.
[34] L. Yu, Yu, J. Song, W. Neiswanger, and S. Ermon. Offline imitation learning with subopti- mal demonstrations via relaxed distribution matching. In Thirty-Seventl AAAI Conference on Artificial Intelligence, 2023.
[35] O. Nachum, Y. Chow, B. Dai, and I.. Li. Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections. 2019.
[36] Zhang, B. Liu, and S. Whiteson. Gradientdice: Rethinking generalized offline estimation of stationary values. 2020.
[37] R. Zhang, B. Dai, L. Li. and D. Schuurmans. Gendice: Generalized offline estimation of stationary values. In International Conference Learning Representations (ICLR), 2019.
10

---

[38] Z. Tang, Y. Feng. L. Li, D. Zhou, and Q. Liu. Doubly robust bias reduction in infinite horizon off-policy estimation. arXiv preprint arXiv: 2019.
[39] Fu, M. Norouzi, O. Nachum, G. Tucker, Z. Wang, A. Novikov, M. Yang, M. R. Zhang,
Y. Chen, A. Kumar, et al. Benchmarks for deep off-policy evaluation. arXiv preprint arXiv:2103. 16596, 2021.
[40] C. Voloshin, N. Jiang and Y. Yue. Minimax model learning. In International Conference on Artificial Intelligence and Statisfics. PMLR, 2021.
[41] M. R. Bruce Hajek. Ece 543: Statistical learning theory, 2018.
[42] P. L. Bartlett and S. Mendelson. Rademacher and gaussian complexities Risk bounds and structural results. In 14th Annual Conference Computational Learning Theory, COLT, 2001.
[43] M. Wulfmeier, I. Posner, and P. Abbeel. Mutual alignment transfer learning. CoRR, 2017.
[44] B. Eysenbach, S. Chaudhari, S. Asawa, S. Levine, and R. Salakhutdinov. Off-dynamics rein- forcement learning: Training for transfer with domain classifiers. 9th International Confer- ence on Learning Representations, ICLR, 2021.
[45] Z. Zhu, K. Lin, and J. Zhou. Transfer learning in deep reinforcement learning: A survey. CoRR, 2020.
[46] C. Packer, K. Gao, J. Kos, P. Krähenbühl, V. Koltun, and D. Song. Assessing generalization in deep reinforcement learning, 2018.
[47] G. Brockman, V. Cheung, L. Pettersson, Schneider, Schulman, J. Tang, and W. Zaremba. Openai gym, 2016.


Reviews:
[{"review_id": "xWue4KJJYQu", "reviewer": "Reviewer_1Zrs", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a method to improve off-policy evaluation from pre-collected data in real world by leveraging access to the simulator. The paper splits the off-policy distribution ratio in real world between $\\pi$  and behavior policy $\\pi_b$ in real world $P_{real}$ by first estimating ratio of $\\pi_b$ in real world $P_{real}$ with $\\pi$ in simulator $P_{sim}$ and then estimating ratio between $\\pi$ in simulator $P_{sim}$ and $\\pi$ in real world $P_{real}$. It is claimed that this two stage procedures uses information from simulator to better do off-policy evaluation and is shown by experiments on 4 simulated tasks and a real robot experiment.", "summary_of_recommendation": "I believe the paper communicates very interesting and relevant idea but needs to improve a lot on writing and presentation. It will make the paper much stronger to have more convincing experiments.\n\nUpdate: Thanks for updating the paper. I believe the paper has interesting ideas that will be relevant to the community and hence I have increased my score.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "I have included the questions in the review above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n1. The paper proposes an interesting setting where by they use the ability to generate lots of data in simulator to aid in off-policy evaluation **given that simulator is close to real world**. This is a relevant problem to robotics community with implications in safety, reliability and guaranteeing policy performance for deployed robots.\n2. The novelty of the work is in combining two different methods and using their strengths to estimate the distribution ratio: a. Ratio between $\\pi_b$ in real world $P_{real}$ with $\\pi$ in simulator $P_{sim}$ can be calculated by sampling method for likelihood estimation as we potentially have access to large number of samples for $\\pi$ in simulator $P_{sim}$. b. ratio between $\\pi$ in simulator $P_{sim}$ and $\\pi$ in real world $P_{real}$ is potentially easier to estimate because simulator is close to real world. \n3. The authors derive theoretical results on their estimator that gives the error bound of estimate that depend on the distribution shift in simulator and real world.\n4. It is shown through simulated and real world experiments that the method outperforms prior method in terms of mean squared error of performance predicted for target policy in real world.\n\nWeaknesses:\n1. Improving paper writing: Paper was not easy to read and the notations are inconsistent and confusing.\na. P,P',P_tr, P_te : It is hard to see why there are so many different notations for dynamics when we are only dealing with 2 different dynamics. This made it very hard to read and understand what is going on in the main paper and proof for Theorem 5.1.\nb. Simple things like denoting P_tr to P_sim and P_te to P_real can make the paper easier to read.\n\n2. Missing details in Experiments\na. How are the experiments set up? What is the dynamics parameter changed to do the experiments? -- These information are key and missing from the main paper.\nb. What is $\\beta$-DICE, over which algorithm is the modification for estimating $\\beta$ made? How are the baselines used here since the baselines cannot leverage simulation information ?\n\n3. Sparse literature review: There has been a lot of work in this domain and some of which might be relevant baselines and seem to be missing in the paper:\na. DARC: https://arxiv.org/abs/2006.13916\nb. https://ojs.aaai.org/index.php/AAAI/article/view/26305\nc. https://arxiv.org/abs/2302.08560\nd. https://arxiv.org/abs/2001.01866 \n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "jsjy1Fo8s9", "reviewer": "Reviewer_dLGh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents an algorithm to evaluate real world performance without the necessity to run the RL policies in the real world. To do so it uses marginalised importance sampling on trajectory data from a simulator as well as from the real world. The whole approach is offline and the results show that their approach is accurate in predicting the real world performance of a given RL policy in sim2sim as well as sim2real tasks. ", "summary_of_recommendation": "the paper is clearly written and easy to follow. The presented work is interesting and focuses on an important problem of sim 2 real transfer. Some details are missing to really asses the applicability of this approach, especially how well this approach generalise to data and trajectories that have not been seen during training. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "see above", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "the paper is concerned with an important topic of evaluating real world performance from offline data.  this would allow to train RL policies offline while keeping in mind the real world performance. The paper is clearly written and the algorithm is easy to understand. The overview image is very helpful as well. The results show that the method can accurately predict real world performance. However the experimental evaluation or the presentation of it is a bit unclear. It would be very interesting to know how much the task that was used to evaluate the methods differs from the offline training data collected in real and in sim. It would also be important to get a sense of how much data is needed as well as what kind of task was evaluated, for example how long and how complicated the task is. this would allow to get a sense for how well the method generalises to out of distribution tasks which would be my main question of the paper.\nI also think the paper could improve by placing itself within the related literature and pointing out the main contributions and distinctions.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "Q7RrDTYgF2", "reviewer": "Reviewer_iwBL", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "In this work, the authors propose a new off-policy evaluation technique, the key breakthrough is to factor the importance sampling into two terms. Occupancy of the target policy in a simulator which can be directly accessed and the ratio of occupancy under simulation and real dynamics. Authors also provide a mathematical bound for the error on such an estimate and show that such a refactoring can be applied to existing OPE algorithms like Dual-DICE.\nComparisons to methods without this estimator for OPE are done in a Sim2Sim setting where transition dynamics are varied for the CartPole and Reacher tasks. Comparisons are made for varying amount of data noise and sim2sim gap and most settings find an improvement over baselines. Improvements are also seen with a real robotics arm reaching task, however, performance drops as the gap between target and behavior policies. ", "summary_of_recommendation": "The new OPE estimator, $\\beta$-DICE, introduced improves over existing MIS methods by removing large density ratio terms and the allowing directly measuring occupancy in simulation. This metric shows improvements in the sim and real evaluations presented in this work. The requirement of a decent simulator adds to the cost of using this algorithm and it is unclear if the improvements warrant it. More robust examples for real world robotics where access to state-actions is scare would make the case a lot more convincing and of interest of a wider robotics audience.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "* Have you tried comparisons to beyond DICE related methods like those described in \"Benchmarks for Deep Off-Policy Evaluation\"?\n* The real world result is very weak since the dataset includes sufficient coverage for state-actions, this almost far from the case in any realistic robotics application. It would be more convincing with a more realistic robotics dataset.\n* The limitations do not address the need from a simulator introduced from this method. And the need for relatively low sim2real gap for the method to be effective. This can in many robotics cases be a large cost.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors motivate a clever refactoring of marginalized importance sampling using a simulator where a term can be directly measured.\n\nStrengths:\n* Clear motivation for their method drawing from the weakness of current MIS\n* Mathematical guarantees on sample complexity and clear explanation of their method\n* Empirical results are clearly presented with error bars.\n* Multiple sim environments, including manipulation and locomotion.\n\nWeakness:\n* Weak baselines since only DICE like methods are considered,\n* Method introduces the need for a simulator with relatively low sim2real gap.\n* Real baseline includes is an \"easy\" scenario for OPE with access to the full state-action space.", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}, {"review_id": "3Zbv8GLiQB", "reviewer": "Reviewer_VF29", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "Offline Policy Evaluation deals with using offline data collected using some (potentially unknown) behavior policy to estimate performance of a different target policy via importance re-weighting. However, even with recent advances in Marginalized Importance Sampling, such methods often produce extremely large density ratios (which makes learning unstable/hard), and work via indirect supervision (which means they perform poorly when offline data is scarce).\n\nThis paper addresses these two issues with the MIS process by proposing a method which breaks the calculation into two factors: Target Policy Occupancy (which has direct supervision) and a second term (which has small magnitude).\n\nThe authors provide theoretical sample complexity guarantees, as well as a sequence of experiments on both Sim2Sim (Cheeta, Cartpole etc.,) and Sim2Real (7DOF Kinova Arm) examples to demonstrate the efficacy of their approach empirically. ", "summary_of_recommendation": "A fairly strong paper with good theoretical backing and extensive experiments. I think presentation could be significantly improved by more careful narrative and explanation in the more mathematical parts, and perhaps some additional analysis to help understand the significance of the experiments.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Mainly covered in the review section: Would appreciate more simplified, english-language explanation of the steps in the sample-complexity proofs, and perhaps more discussion of the results of the experiments.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper makes a strong technical contribution in the form of the advancement of the MIS methodology of offline policy evaluation in robotics. It has a theoretical component which gives the merits of the algorithm in terms of sample complexity guarantees. It also has extensive experiments on Sim2Sim with ablations for data-collection, target policies and environments, as well as real world Sim2Real experiments on a robotic arm.\n\nMy main struggle with this paper were that the mathematical sections were extremely dense, and I found many of the results difficult to follow. For example, Theorem 5.1 is huge, but little description is given to the intuitive significance of each component other than terse definitions of each symbol. I think this paper would be vastly improved by small additions of english-language explanations of each step, and perhaps even mini-summaries briefly, in laymans terms, of what these results actually mean for performance. Section 6 is similar, launching into an optimization derivation \"exploiting Fenchel Duality\" without giving context as to what we are doing here.\n\nThe related work section does a good job placing the work in context of recent MIS work. However, at around 15-lines, I found it to be quite terse and limited in scope. It briefly mentions that model-based methods also exist, but that's about it. I wonder if it could be placed in the broader context of other approaches to long-horizon / high-variance sampling problems. For example, Defensive/Multiple Importance Sampling, & Stratified / Multi-Level Adaptive Splitting?\n\nSimilarly, I found the Experiments section to be quite short. The authors have clearly done extensive experimentation, both in the main paper and supplementary material, but I think the paper could be improved by a more detailed analysis/explanation of the results. For example, their main observation is that their method \"is comfortably able to outperform...baselines across a range of target policies\", and that \"as the sim2sim gap increases, the performance drops significantly\". Could you expand on the \"why\"? Additionally, with the line graphs in Figure 2, it actually looks like the baselines may be trending downwards, while Beta-DICE is trending upwards in terms of error. Also for the \"length\" graph, there is a massive drop in the \"simulator\" line at 0.0 before recovery. What do these anomalies mean?\n\n### Minor Issues \n\n- Line 248: \"Prediction error decreases\" -> \"increases\"?\n- 262: \"Framwork\" -> \"Framework\"", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses the problem of off-policy policy evaluation, namely evaluating a policy without running it on the real environment, but based on offline data collected from some other behavior policy in the environment. Most existing techniques use some variant of importance sampling to evaluate the policy based on offline data, but this causes estimation error if the offline transitions are too far from the policy's transitions. To address some of these estimation issues with importance sampling resulting partially from distribution mismatch, the paper assumes access to a simulator with low sim-to-real gap, to try to limit this mismatch. Using samples from the simulator has another advantage: it allows computing the importance sampling factor via simulator samples. The paper provides a sample complexity guarantee by upper bounding the evaluation error by 1/sqrt(n), where n is the number of samples in the offline dataset, and also the number of simulator evaluations.  

Reviewers were concerned about the breadth of environments and simulated robots this method was evaluated on. Most of the examples in the evaluation section are sim-to-sim with controllable noise levels. The paper has only one scenario of offline data from a real robot, namely a Kinova arm. The authors performed additional experiments to mitigate some of these concerns. 

Looking at previous papers in the off-policy evaluation literature, my impression is that a limited number of sim-to-sim scenarios is the norm. While I agree with the concerns that more real data would have made the conclusions of the paper more reliable and robust, the idea of the paper is interesting and relevant for robotics, especially in cases where the sim-to-real gap is small. 

I think it is worth accepting this paper as poster.
Decision: Accept (Poster)