Conservative Evaluation of Offline Policy Learning
Authors: Hager Radi Abdelwahed
Josiah P. Hanna
Matthew E. Taylor
Venue: Accepted by TMLR ()

Abstract: The world offers unprecedented amounts of data in real-world domains, from which we can develop successful decision-making systems. It is possible for reinforcement learning (RL) to learn control policies offline from such data but challenging to deploy an agent during learning in safety-critical domains. Offline RL learns from historical data without access to an environment. Therefore, we need a methodology for estimating how a newly-learned agent will perform when deployed in the real environment \emph{before} actually deploying it. To achieve this, we propose a framework for conservative evaluation of offline policy learning (CEOPL). We focus on being conservative so that the probability that our agent performs below a baseline is approximately $\delta$, where $\delta$ specifies how much risk we are willing to accept. In our setting, we assume access to a data stream, split into a train-set to learn an offline policy, and a test-set to estimate a lower-bound on the offline policy using off-policy evaluation with bootstrap confidence intervals. A lower-bound estimate allows us to decide when to deploy our learned policy with minimal risk of overestimation. We demonstrate CEOPL on a range of tasks as well as real-world medical data.

Published in Transactions on Machine Learning Research (9/2024)
Conservative Evaluation of Offline Policy Learning
Hager Radi Abdelwahed
vudi@raalerraa.cu
Deportment of Computing Science
University of Alberta, Edmonton, Canada
Josiah P. Hanna
jphanna Scs wisc edu
Computer Sciences Department
University of Wisconsin Madison
Matthew E. Taylor
matthew. e. www.or@rulberto..co
Department of Computing Science Alberta Machine Intelligence Institule (Amii) Universily of Alberta, Edmonton, Canada
Reviewed on OpenReview: https: wttpr//vepprrvee....mmmm net/ forum? id= KLoOTK
Abstract
The world offers umprecedented amounts of data in real-world domains, from which can develop successful decision-making systems. It is possible for reinforcement lcarning (RL) to learn control policies oflline from such data but challenging to deploy an agent during learning in safety-critical domains. Offine RL learns from historical data without access to an environment. Therefore, we need a methodology for estimating how a newly-learned agent will perform when deployed in the real environment before actually deploying To achieve this, we propose a framework for conservative evaluation of offline policy learning (CEOPL). We focus being conservative so that the probability that our agent performs below baseline is approximately 6, where 8 specifies how much risk we are willing to accept. In our setting, we assume access to a data stream, split into a train-set to learn an offline policy, and a test-set to estimate a lower-bound on the offline policy using off-policy evaluation with bootstrap confidence intervals. lower-bound estimate allows us to decide when deploy our learned policy with minimal risk of overestimation, We demonstrate CEOPL on a range of tasks as well real-world medical data.
1 Introduction
Suppose someone else is controlling sequential decision making task for you. This could be a person trading stocks, hand-coded controller for chemical plant, even PID controller for temperature regulation. Offline reinforcement learning (RL) allows us to learn policies from historical data collected by some other controller. But when would one want to switch from the existing controller to the new policy? This decision may depend on the cost for continued data collection from the existing controller, your risk appetite, and your confidence in the performance of the policy you have learned. This paper takes a critical step towards the question: how (confidently) sclect the right time deploy when Icarning offline, if we do not have access to the environment nor the policy generating the data? Offline reinforcement learning is way to train off-policy algorithms using existing data. It presents a great opportunity for learning data-driven policies without environment interaction. In safety-critical applications such as in healthcare or autonomous driving, there a large amount of data that we can use to learn RL policies and hence use for decision making (Gottesman et al. 2019). Learning with offline data challenging because of the distribution mismatch between the data collected by the behavior policy that collected the data, and the offline agent, which learns from data (Levine et al., 22020). What is even more challenging is evaluating offline agents in the offline RT setting if we assume access the environment; in some

---

Published in Transactions on Machine Learning Research (9/2024)
domains, we cannot exccute our learned policy until it is good enough because it can be costly dangerous if it performs worse than the current policy or controller. This limitation raises the possibility of using policy evaluation (OPE) methods, where we can estimate the value of a policy using trajectories from another policy, to predict what the performance of the offinne agent is at any point of time during training. better to use OPE with confidence intervals to control the risk overest the policy's performance, which referred to as conservative off-policy evaluation (COPE).
We further investigate
We present framework for conservative evaluation for offine policy learning (CEOPL). CEOPL tack- les the setup where combine offline policy learning with conservative off off-policy evaluation; we learn policy from pre-existing data and evaluate it to provide a confidence lower-bound estimate on its re- turn. The goal is to learn a target policy purely from data and evaluate its performance while learning so that we can tell when it ready for deployment. In conservative off-policy evaluation (COPE), we ensure safety by using bootstrap confidence intervals to provide a lower-bound on the policy value esti- mates using a reasonable amount of test data. COPE provides an approximate solution to high confi- dence off-policy evaluation (HCOPE); HCOPE estimates lower bound on the policy's value that is correct with set level of confidence, but requires substantial amount of data to achieve tight lower bound. Our target sctup is summarized in Figure ! we have some source of data, from which we request data sam- ples. At each step, data is split into training and test- ing. After each offline learning step, we test the pol- icy using conservative evaluation. A policy passes the safety test. if lower-bound the policy's 's value is bet- ter than the value of the data distribution. We con- tinue the process of training/ Seaining/esstng lesting for few itera- tions until the testing shows the policy can outperform the existing controller with a confidence level 5. We dynamically receive samples, continuously perform RL updates, and continuously monitor confidence interval on the changing policy until it reaches a sufficient level for deployment. We hypothesize that our conserva- live off-policy evaluation (COPE) is preferred over off- policy evaluation (OPE) because OPE may be prone to overestimation, which is problematic for safety-critical problems. CEOPL acts as a workflow for offline learn- ing and evaluation in safety-critical domains. Contri- butions of this article are summarized as follows:
source
Train agent
Deployment
False Safety test for True
agent?
Figure 1: Sctup for continual safety- evaluation of offline RL: this figure illustrates the loop of interaction between an offline agent and data while the agent is trained, evaluated and de- ployed if it passes the safety test.
CEOPL, framework combining offline RL with conservative off-policy evaluation for both discrete and continuous control tasks.
Studying the feasibility of oflline evaluation without access to an environment nor the behavior policy.
Exploring the effect of offline learning on OPE methods given constantiy-improving target policy (as opposed to a fixed policy, as previously studied in the literature).
Demoustrating CEOPL on real-world medical data.
2
Related Work
In this section, we provide a review of the current literature of offline reinforcement learning and off-policy evaluation. We also discuss the gap in research our work is trying to fill.
Offline Reinforcement Learning Offline reinforcement learning refers to RL algorithms learning from a fixed dataset D without environment interaction further data collection. Such data is the transition tuples
2

---

Published in Transactions on Machine Learning Research (9/2024)
(st, de, 811,, 7411) resulting from historical agent-environment interaction, which will eventually compose a trajectory H of size L. We learn no; new target policy that is not necessarily similar to the behavior policy Th. In this work, we assume we have access to a dataset D = {HH, H2,,..,, Ha) as a set of n trajectories generated by an unknown behavior policy Tb.
For offline RI, a survey paper (Levine et al. 2020 categorizes offline RT algorithms into model-free methods and model-based methods. Our paper only considers model-free approaches for offline RL, where model-free methods are sub-categorized into policy constraint methods that constrain the learned policy to be close to the behavior policy and uncertainty- values to reduce distributional shift. In policy gradient methods, a policy constraint can be enforced directly on the actor update to keep the new policy as similar as possible to the behavior policy while learning. This constraint can be implicit KL-divergence, which does not model the behavior policy; for example, see advantage- weightte regression (AWR) (Peng et al. 2019) or advantage weighted actor-critic (AWAC) (Nair et al. 2020). A constraint can also be an explicit divergence constraint that require an estimation for the behavior policy as done in the way off-policy (WOP) algorithm (Jaques et al. 2019). In practice, policy constraint methods far seem to outperform pure uncertainty-based methods, as shown with the batch constrained deep Q-lcarning paper (Fujimoto et al. 2019b). In bootstrapping error accumulation reduction (BEAR) (Kumar et al.) 2019), authors identify bootstrapping error as the source of instability of offline RL methods; they propose doing distributiion-constraneed backups via maximum-mean discrepancy (MMD) (Kunar et al.) 2 In belhavior-regularizeed offline reinforcement learning (Wu et all (2019). a general framework, behavior constrained actor-critic (BRAC), is introduced to cover different ways of regularization to offline policies, whether as value penalty or policy penalty. Non-constrained methods include traditional Q-lcarning (Watkins & Dayan 1992), double DQN (Hasselt et al., 2016), and soft actor critic (SAC) (Haarnoja et al., 2018). which are not always successful in the fully oflline setup. Behavioral cloning (BC) (Bain & Sammut 1999). as main method for imitation learning, is another way for learning offline policies from historical data; BC proved to perform well compared to offline RL. methods under medium-quality and expert data (Kumar et al. 2019).
methods that attempt to estimate the epistemic uncertainty of Q-
Off-policy Evaluation Off-policy evaluation (OPE) methods evaluate a target policy vo using data gen- erated by a behavior policy Tb. When the off-policy estimate is guaranteed with some confidence that its performance not worse than the behavior policy To. it is referred to as high confidence off-policy policy evaluation (HCOPF) (Thomas et al., 2015a). An empirical study of OPE methods (Voloshin et al.] 2019) discussed the applicability of both methods and presented method selection guidelines depending environment parameters and the mismatch between no and Th. This study categorized OPE methods into importance sampling (IS) methods, direct methods, and hybrid methods that combine aspects from both IS and direct methods. Importance Sampling (IS), or Inverse Propensity Scoring, as it is known in statistios, is one of the most widely used meth for off-policy evaluation where rewards are re-whighted by the ratio between Ty and лe (Precup et al.] 2000). This weighting results a consistent and unbiased off-policy estimator. Later versions of importance sampling provide lower-variance estimates such as weighted impor- tance sampling (WIS) (Mahmood et al. 2014), per-decision importance sampling (PDIS), and per-decision weighted importance sampling (PDWTS) (Precup et al., 2000). To avoid bias due to using the off-policy state distribution and the high variance of importance weights, Dice-style methods (Nachum et al. 2019 9a)
the
(Nachum et al.) 2019b) (Zhang et al.) 2020) (Yang et al.) 2020 estimate the marginal importance ratio between state density distributions instead. Secondly, direct methods rely on regression techniques to di- rectly estimate the value function of the policy without access to the behavior policy. This category includes model-based methods where the transition dynamics and reward are estimated from historical data via a model. Then, the off-policy value is computed with Monte-Carlo policy evaluation in the model (Hanna) et al. 2017). To achieve conservative OPE, MB-bootstrap (Hanna et al. 2017) uses model- off-policy estimator with bootstrapping to construct confidence intervals for OPE estimates. A recent work introduced HAMBO (Rothfuss el al. 2023), which uses an uncertainty- learned model of the transition dynamics to obtain tight lower bounds of policies. Another direct method is fitted Q-evaluation (FQE) (Le et al. 2019), a model-free approach that acts as the policy evaluation counter-part to batch Q-learning or fitted Q-iteration (FQI) [Riedmiller] (005). The last category contains the hybrid methods that combine different features from IS and direct methods, which mainly involves doubly-robust. methods (Jiang & T.i 2016) that
3

---

Published in Transactions on Machine Learning Research (9/2024)
use direct model to lower IS variance. A few other methods improve upon this approach, such as weighted doubly-robust (WDR) (Thomas & Brunskill 2016), and MAGIC (Thomas & Brunskill 2016).
Gap in The Literature In the offiine RL literature, there is no standard way to evaluate a policy while learning offine; most of the literature still evaluates the performance offline algorithms in the environment or in a simulator, by running policy online. workflow for offline RL (Kumar et al. 2021) proposed a set of metries to indicate how the algorithm can be further improved, such detecting over-fitting and under- fitting, providing clear workflow for estimating the policy performance. At the intersection of offline RL and OPE, there has been recent work combining these two areas. One paper discussed the applicability of FQE [Le et al., 2019) to test the performance of offline policies under different paper-parameters and select the best performance (Paine et al., 2020). Active oflline policy selection (Konyushova et al. 2021 uses OPE to warm-start the evaluation process and choose which policy to evaluate online when given a limited budget of online interactions. In benchmarks for deep off-policy evaluation (Fu et al.] [021), authors evaluato offiine policies learned on continuous control tasks using various OPE methods ranging from importance
sampling, model-based methods, and doubly-robust estimators. They show how challenging OPE can be evaluation method (Fu et al.) 2021).
Even considering the previous work combining offiin RL with OPE (Paine et al. 2020 Fu et al. 2021
none discussed the feasibility of evaluating offline RT. agents as we learn no. and how much we can trust OPE as an approach for testing. The setup we are studying is quite different from the current literature because previous work for off-policy evaluation assumed behavior policy मe and a target policy Tp, where both policies are fixed and may be related. For example, in a study for high-confidence OPE methods (Thomas et al., 2015a), the target policy is initialised as subset of the behavior policy such that they are close to each other. In another study for safe improvement (Thomas et al.) 2015b), the Daedulus algorithm learns safe target policy as a contimuous improvement over the behavior policy. This is quite relevant to our work; however, it is not purely offline learning because their algorithm uses data from an older version of the policy it currently improves to perform both the improvement step and safety tests. With the doubly-robust estimator, results are presented with different versions of my such that my is always a mixture of my and wy with different degrees to ensure their relevance (Jiang & Li 2016
Recent work on offline RL focuses on continuous control in near-complex domains and environments [Levine et al.) 2020). As detailed above, OPE liter- ature focuses on much simpler domains in discrete and continuous control [Voloshin et al. 2019 None of the previous work discussed the fcasibility of us- ing conservative OPE to evaluate offline RL policies. We believe the setup are tackling is under-studied in the literature where the target policy is not nec- essarily similar to the data collection policy, which is the case for ofline RL.
Dj
D Dirain
Dtest
Training butter
Conservative off-policy
Offinee RL optimization
3 Methodology
In this section, we introduce our framework CEOPL in detail with the methods used for both learning and evaluation of an offline policy. CEOPL couples offline RL algorithms with conservative off-policy evaluation as a feasible evaluation method for of- fline agents in safety-critical domains. Conserva- live evaluation is defined by any ofl-policy evalu- ation method which seeks a broaabrobybylity lower bound on the policy's estimate. In CEOPL, we sam- ple data from a buffer of data dynamically at each iteration, perform policy updates, and continuously
Request
the policy good enough?
Yes, Deploy!
No
Figure 2: Proposed workflow for CEOPL: this figure shows the methodology to follow, when given source of data, to both learn policy offline and estimate its value confidently.
4

---

Published in Transactions on Machine Learning Research (9/2024)
monitor its performance with confidence interval
until it reaches a reasonable performance and is ready to be deployed. The policy learning offline does not interact with the environment unless it passes the safety test. Moreover, we assumne no control or access to the behavior policy т producing the data, but we assume that we can request more data as long as our oflline policy does not. pass the safety test. However, the same methodology applies we just have access to static datasct of trajectories (which we demonstrate in Section G We have iterations where, each iteration i, dataset of trajectories D, of size n is requested from the main data source. D, is split between Dirain to train our offiine policy Te, and Dtest to perform the safety test using conservative off-policy eval- uation methods. Dtrain is accumulated over iterations in the training data buffer while a new set of Drest is used each evaluation iteration for the confidence interval to hold. CEOPL terminates when the learned policy is ready to be deployed with appropriate confidence. Instantiating CEOPL requires selecting an oflline policy learning method and an off-po evaluation method for computing off off-policy estimates. We study various methods for offiine policy learning and evaluation as our methodology. Figure / shows an overview of CEOPL framework based on given data source, where the interaction between an agent and the data source is summarized as follows:
1. An agent receives a set of data D; without access to the behavior policy.
2. We optimize a policy Te offline using Derain with an offline RL algorithm (Section 3.1
3. We perform a safety test using Drest with conservative off-policy evaluation method (Section 3.2)
4. Our new policy is either ready to be deployed or we go back to step 1.
Algorithm Consorvative Evaluation for Offline Policy Learning (CEOPL) Framework
Input: initial тe, dataset D of m trajectories, confidence level 5 E [0, 1], number of trajectories needed every iteration n, number of bootstrap estimates B, offline learning method $, and OPE method v Output: Ta, (s(TT)): 1 1 8 lower-bound on ((Te)
1: Let =
while (TT) S ((T)) do Request set of trajectories Dj with size
Split D; into Drrain, and Diest
5: Drrain = Dtrain U Dtraine
6: Optimize policy To. = 4 (DDtrin))
7: Evaluate policy (G(TT)) = BCT(TO, Drast, 6, B, V)
8: end while
return TT. (TT)
We refer to each offline policy learning method with & and each off-policy evaluation method with V. CEOPL is further explained in Algorithm [ The inputs are a dataset of trajectories D, a confidence level 8 appropriate for the problem in hand, offline policy learning method + and an off-policy evaluation method V to combine with bootstrap confidence intervals, shown Algorithm 2 in Appendix A.3 The output will a policy Irained offline my and a coufidence lower-bound est isttee of its return és (m²)) Since the stopping condition in Algorithm [ is not always satisfied if no evaluation method can detect an improved policy, we set maximum number of itcrations for training and evaluation in our experiments.
3.1 Methodology: Offline Reinforcement Learning
Is it possible to learn a good policy oflline given data and no environment interaction? Earlier work (Agarwal let al. 2020) tried to answer this question, and showed how capable RL algorithms are in the offline setting, since they were able learn a successful DQN agent totally offline when provided with diverse data. However, benchmark on batch RL (Fujimoto et al. 2019a) showed how RL without correction can fail in the offline setting given ordinary data.
5

---

the target Published in Transactions on Machine Learning Research (9/2024)
Table 1: Categories for offline learning: this table shows different categories for offline learning methods for both discrete and continuous control, with examples of algorithms used in our experiments.
Discrete Control
Continuous Control
Off-policy
Double DQN (Hasselt ct al. 2016)
SAC (Haarnoja al. 2018
Imitation Learning
Behavioral Cloning (Bain & Sammul 1999)
Behavioral Cloning
Offine
BCQ (Fujimoto et al. 2019b)
BCQ & BEAR (Kumar al. 2019
In CEOPL, we investigate different techniques for learning from offline data, ranging from normal off off-policy RL algorithms (off-policy), imitation lcarning techniques (imitation learning), and policy constraint methods specific for offline learning (offline). Imitation learning is a form of supervised learning and is good at learning from expert data or demonstrations. Policy constraint methods are designed to mitigate the challenges for normal off-poliey to learn offline, such as bootstrapping error (Levine et al. 2020). This categorization is inspired by the offiine RL tutorial Levine et al., 2020) and is shown in Table Details about each algorithm are further explained in Appendix A. Given the data samples used from data source, we aim to learn a policy totally offline without environment interaction or access to the policy generating the data. This offline policy expected to be at least as good as the behavior policy in case of imitation lcarning and better in the case of other algorithms,, after few iterations.
3.2 Methodology: Conservative Off-policy Evaluation
Conservative off-policy evaluation extends OPE methods to lower-bounding the performance of policy, no, when combined with bootstrapping confidence intervals.
Off-Policy Evaluation
While a policy is being improved offline, we interested in evaluating such policy. Off-policy evaluation is a promising technique to evaluate policy that is continuously learning given access to fixed data. An off- off-policy estimator is a method for computing an estimate t(TT) for the true value of the target policy u(To) using trajectories D collected while following another policy Th, which is what OPE methods do. When evaluating, we use new samples for testing each time to avoid the multi-compprisonn problen This sampling also to ensure that we not over-fit our OPE estimates or tune training parameters to reduce the estimate error. Table T shows the three categories of OPE methods, with examples under each category. The methods in bold are the ones we use for evaluation. For instance, we use Weighted Importance Sampling (WIS) [Mahmood et al. 2014., Direct Model-based (MB) (Hanna et al. 2017 and Weighted Doubly-Robust (WDR) Thomas & Brunskill 2016) estimators as shown. More details are further shown in Appendix A.2
Table 2: Different OPE mothods categorized: This table shows the three main categorics of OPE methods, importance sampling, direct methods and hybrid methods with examples with the methods used in our experimentation.
Importance Sampling
WIS Maahmodd et al. 2014)
Direct
Model-based (Hanna et al. 2017
Hybrid
Weighted doubly-robust/ www.-oooosstohsasssss Thomas & Brunskill 2016)
Bootstrapping
To conservatively evaluate in CEOPL, each OPE method is combined with bootstrapping to approximate confidence lower bound of (T))) on ((Te) such that ()()))) S ((TT)) with probability at least 1- 8. In principle, bootstrapping can be replaced by any other method that provides tight lower bounds. Consider data sample D of n random variables Hj for = 1,2,.., ..., n where we sample H, from some i.i.d.
2The problem occurs when conducting multiple statistical tests simultaneously with reusing data, so the confidence bound does not strictly hold anymore.

---

Published in Transactions on Machine Learning Research (9/2024)
distribution of data. From the sample of data D, we can compute sample estimate â of parameter z such that 2 = I(D) where f is any function that estimates x. Given a dataset D, we create B resamples with replacement, where B is the number of bootstrap resamples, and compute an estimate for x, 2, on each of these resamples. Bootstrapping (Efron 1979) allows us to cstimate the distribution of i with confidence bounds. The estimates compuled with different resamples will determine the - 6 confidence interval. In our sctup, the parameter of interest x is the expected return of a policy v((o))
With a confidence level 8 E [0.1] and B resamples of the dataset of trajectories D, we use bootstrapping to approximate
bouur- confidence interval of ((T))) on v(Te) such that vs(Te) S v(TT)) with probability at least 1 - 5. This bootstrap method is referred to as the percentile bootstrap for confidence intervals CCarpentrr & Bithell 2000). After obtaining B resamples of the dataset, we compute ((TT)) with each of these resamples. Then, all the off-policy estimates ((TT)) are sorted ascendingly and the index x B is chosen to calculate (x)) When bootstrapping is combined with off-policy evaluation methods, we can estimate the distribution of v(TT), and use it estimate 0g (TT) with probability at least 1-8. We refer to the algorithm used as bootstrap confidence intervals (BCT) and the pseudo-code is detailed in Algorithm 2 in Appendix A.3 Although bootstrap confidence intervals provide approximate high-confidence estimates, they provide a practical approach with small amounts of data compared to exact concentration incqualities that may be too loose to use.
4 Experiments
This section discusses the experiments and the results of CEOPL on simulated setups in discrete and con- tinuous control tasks. In our experiments, two simulated environments (MountainCar-vy and Pendulum-vO) were used to demonstrate CEOPL in discrete and continuous control, respectively. This choice shows how the complexity of the environment, state and action dimensions, and horizon are affecting offline evaluation. To mimic the setup where there is source of trajectories without access to the environment itself, we simu- late a source of data in each environment Such data resulting from a policy that is not optimal nor random resembles real-world data and leaves room for improvement when training offline RL policies. For each environment, a policy is optimized offline using three different offline learning methods and evaluated simultaneously using three different off- off-policy evalua- tion methods. Following Algorithm [1 in each iteration, we get m trajectorics from our data source which we split between training and testing. We use more data for testing than training since CEOPL requires large amounts of data to estimate tight bounds. For cach improvement method, we load the train split into the training buffer and do k training epochs where & varies among oflline RL algorithms. Since we exceptionally have access to the true environment in these simulated domains, we compute the true value of the offline policy by testing it in the actual environment and use it for comparison. For bootstrapping, we use 5 = 0.05 to get a 95% confidence lower- bound using B = 2000 bootstrap estimates, as recommended by practitioners (Efron) 1979). In weighted doubly-robust with boolstrapping 3.2 we use a value of B = 224 for the number of bootstrap estimates to avoid heavy computation; this can still get us good approximation as suggested by MAGIC (Thomas & Brunskill 2016). Since we have no access to Th, we estimate A given the test data 2021 with behavior cloning model that is improved gradually with more test data.
partially-trained policy that collects data of medium quality,
(Hanna et al.
Results on MountainCar-vO: For each iteration, 300 trajectories are sampled, where 20 trajectories are used into the training buffer and 280 trajectories are used for evaluation. This totals to 3000 trajectories over the 10 iterations. Each evaluation iteration includes performing & training epochs to the offline policy; a training epoch is a single optimization step of the policy over batch of data. k is set to be 29 for BCQ and Double DQN, while BC does 26 policy updates in one iteration (since BC was faster to optimize). Figure g shows how different offline policy improvement methods perform given medium-quality data with conservative evaluation, indicating which method can tell when (s(m)) > ((z,). While the x-axis shows the number of evaluation iterations, the y-axis shows the value estimate of a policy across different itcrations. Behavior cloning, as an oflline policy improvement method, can only perform as well as Th- Double DQN and BCQ were able to outperform To. The true value of a target policy is calculated as the average return when running the policy in the actual environment (not possiblc in practice) for 1000 episodes. All reported results are an average of 40 runs, while the shaded area shows the standard error. The value of the behavioral policy, ((m)), is the sum of undiscounted rewards of all trajectories in the data set.
7

---

Published in Transactions on Machine Learning Research (9/2024)
Ng: Behavior Cloning (BC)
Rgi Batch-constrained Q-learning (BCQ)
-20
-40
440
-10
-70
80
-90
90
-100
10
2
Evaluation Iter ations
Evaluation Iterations
Te: Double DQN
-20
-40-
((TT))
((m))
V(TTe)
-60-
Vs((T) WIS
-80-
Vs((Te) MB
Vo([Te) WDR
100
10
Evaluation Iterations
MorntainCar-v0. this figure shows the performance of the target policy as it is training offline over iterations (x-axis). Weighted importance sampling (WIS) failed to detect that the offline policy outperforms Th. The model-based estimator (MB) and weighted doubly-robust (WDR) were able to determine when an offline agent outperforms лy. While the y-axis shows the estimated policy value ((T), we show the empirical estimate ((m) for the behavioral policy (given the dataset), and the conservative off-policy estimate (T8))) across different estimators.
Figure 3: CEOPL
For conservative evaluation of each offline agent, weighted importance sampling (WIS) with bootstrapping failed to detect that the offline policy outperforms He for all learning methods. The model-based estimator (MB) and weighted doubly-robust with bootstrapping (WDR) have much less error with respect the true value of policy and were able infer when an offline agent outperforms To and hence ready to be deployed. For instance, with 95% confidence, in the case of offline improvement with Double DQN, were able to tell that our new target policy is better than the behavior policy at iteration 4 using two different istiors (MB and WDR).
Results on Pendulum-vO: For each iteration, 500 trajectories are sampled, where 100 trajectories goes into the training buffer and 400 trajectories for evaluation. This totals 5000 trajectories over the 10 iterations. Figure shows how different offline policy improvement methods perform given medium-quality data with conservative evaluation, indicating which method can tell when (T())) > 6(m)). Behavioral cloning as an offline policy improvement method can only perform well the data by Ta. while BEAR and BCQ were able to outperform тo. We did not report results SAC (Haarnoja et al. 2018) since it failed to learn a policy better than the behavior policy. In MountainCar-v0,, the true value of a target policy is calculated as the average return when running the policy in the actual environment. (which is not possible in practice) for 1000 episodes. All reported results are an average 40 runs, while the shaded area shows the standard error. ((m)) the sum of undiscounted rewards of the data set, which represents the value of the behavioral policy. Similar to the results on MomtainCar-vv., WIS with bootstrapping failed detect that the offline policy putperfomm Te- The MB estimator achieve very tight lower-bound with the true value of policy
8

---

Published in Transactions on Machine Learning Research (9/2024)
Ng: Behavior Cloning (BC)
Tg: Batch Constrained Q-Learning (BCQ)
-400
-400
V(R)
V(r)
-500
-600
-000
1100
-1200
Evalustion iterations
Evaluation Iterations
To.: Bootstrapping
Accumulation Reduction (BEAR)
-100
((T,)
-600
v(TTe)
(o((Te) WIS
vo((T)) MB
(o((Te) WDR
-1200
10
Evaluation Rerations
Figure 4: CEOPL Pendulum-vO: this figure shows the performance of the target policy as it is training offline over iterations (x-axis). Weighted importance sampling (WIS) failed to detect that the offline policy outperforms Tg. The model-based estimator (MB) and weighted doubly-robust (WDR) were able to determine when an offline agent outperforms иy, While the y-axis shows the estimated policy value ((T), we show the empirical estimate imm) for the behavioral policy (given the dataset), and the conservative off- off-policy estimate (Te) across different estimators.
and were able to identify when an offline agent outperforms ma and therefore is ready to be deployed. WDR estimates are also affected by the improvement methods as in the case with WIS.
5
Analysis & discussion
5.1
Overestimation of OPE vs. Conservative OPE:
It is important to question whether conservative evaluation in our proposed framework is worth the added complexity and computation of bootstrapping over the vanilla off-policy evaluation (OPE). Although both OPE and conservative OPE can provide value estimates of an offline target policy, conservative OPE lower- bound estimates underestimate the value of the offline policy which serves our purpose of ensuring safety. Moreover, most existing OPE estimators are prone overestimating the value of the policy Thomas et al.) (2015a). Empirical results on simulated environments showed how OPE estimates overestimate the true value of the offline policy, shown in Figure 5 Even if not all OPE estimators value of the offline policy in Figure 5 there are no guarantees on the performance of the reported OPE estimators. On the other hand, lower bounds in CEOPL are guaranteed to only overeslinate the true value of the offline policy within the allowable 5% error rate for a 95% confidence, as shown in Figure] 7 in Appendix B.2 Bootstrapping has strong guarantees the size test data goes oo, but lacks guarantoes for finite samples (Hanna al.] 2017); this is because it assumes that the bootstrap distribution representative of the true distribution of the statistic of interest. As result of this assumption, bootstrapping is considered
overestimating the true

---

Published in Transactions on Machine Learning Research (9/2024)
Off-policy (Double DQN)
Imitation Learning (BC)
Offline (BCQ)
WIS
49.032 + 0.57
28.742 + 1.22
43.66 + 0.84
MB
7.85 + 0.72
22.716 + 1.09
13.645 + 0.65
WDR
6.18 = 1.45
17.642 + 2.79
12.81 + 2.42
Table 3: Mean error between CEOPL estimates and true value of To MountainCar-vO: this table shows the mean error of conservative OPE estimates, 66 (TT). over all training iterations and multiple runs given each offline improvement method. Lowest mean errors for each method are shown in bold., and standard error over 40 runs is reported.
Imilation Learning (BC)
Offine (BCQ)
Offiine (BEAR)
WIS
268.5 + 28.93
602.49 78.05
432.22 i 47.5
MB
33.48 + 0.77
25.02 +: 2.5
19.0 + 5.1
WDR
161.1 + 13.47
307.11 + 37.3
54.75 + 10.43
Table 4: Mcan error between CEOPL estimates and truc value of Ty on Pendulum-vO: this table shows the mean error of conservative OPE estimates, (i(())), over all training iterations and multiple runs given each offline learning method. Towest mean errors for each method are shown in bold, and standard error over 40 runs is reported. SAC is not reported as it failed to learn good policy.
semi-safe because the assumption it makes may be false. Prior work Kostrikov & Nachum (020 identified conditions, such as sufficient data size and sufficient coverage, under which statistical bootstrapping is guarantced to yield correct confidence intervals. Other work (Morrison et al. (007) shows that bootstrapping is still safe enough for high-risk predictions with known record of producing accurate confidence intervals. Therefore, it is better to rely CEOPL for conservative evaluation in safety-critical applications.
5.2 What affects CEOPL?
If we are to apply CEOPL to any problem, OPE methods and offinne RL methods are different factors affecting the performance. To investigate this, we do further analysis and report total variation (TV) distance, mcasure of similarity between two probability distributions; this is to check how the learning method of an oflline policy is affecting TV distance between íi and and hence affecting the off-policy estimates. Results are shown in Figure g in Appendix [B.]] where the total variation distance between the
offline policy тe and the estimated behavior policy i. across the different offline learning methods is reported. For discrete control, behavioral cloning (imitation learning) can achieve a much lower distance than other improvement. methods (batch-constraiined Q-learning and double DQN) whether they are constrained or non-constrained. This is because behavioral cloning forces its target policy to be close to the behavior policy while other methods do not. The same applics to contimuous control in Pendulum-v0; behavior cloning achieves the lowest error between H. and πe. This insight on divergence explains why weighted importance sampling achieves the lowest error between the estimate and the true value in the case of behavioral cloning, as shown in Tables 3 and 4 the error grows for other methods that do not constrain the policy to be close to the data distribution. This analysis suggests that the choice of a OPE method is dependant on the offline method chosen to optimize the policy. Other factors affecting the performance include the horizon of the trajectory and the environment dynamics, which we discuss further in Appendix B.3
the performance of conservative off- off-policy evaluation, and OPE in general can be affected by the following factors: a) divergence between Ån and тo discussed earlier in Section 5.1 b) the horizon of the trajectory, c) the environment dynamics. The horizon of the trajectory affects importance sampling such that it suffers from high variance with longer horizons (Liu el al., 2020). The environment dynamics also affect direct and hybrid methods. The MB estimator performs well when is possible to model the environment dynamics such as the case for both MountainCar-vo and Pendulum-vO environments. The WDR estimator also performs well because q-values can get more accurate when is easy to learn a model of the environment. However, the WDR estimator is still affected by the divergence between âs and my
Takeaways:
10

---

Published in Transactions on Machine Learning Research (9/2024)
in the case for BC and BCQ algorithms. WDR estimates get worse the divergence increases. As result, we can conclude that
When it is possible to model the environment dynamics, we should rely on estimators that do not take my into account (e.g., direct methods or hybrid methods) so that value estimates are less affectod by the divergence between To and Ab (as the case for WIS).
In short-horizon environments, we should rely on importance sampling methods when the offline policy algorithm is constrained to be similar to ft..
Based on our empirical analysis in subsections 5.1 and 5.2 we conclude that conservative off-policy estimators are considered a safe evaluation method for offline learning with enough confidence that minimizes the risk of overestimating the true performance an offline policy. In contrast, off-policy evaluation suffers from overestimation to the true values so makes sense rely on conservative evaluation in CEOPL instead, for domains where safely essential. Further, our ability to accurately detect an improved policy with CEOPL is function of different factors: the offline learning algorithm, the environment dynamics and the trajectorics horizon.
6 Case Study: Real world data
Behavior Cloning (BC)
-10
In this section, we demonstrate our proposed frame- work CEOPL on weal-world medical data to see how our framework can perform.
V(ro)-WDRR
30
v(ng)
v(m))
400
6.1 Problem: Sepsis Treatment
((m)
Vnnol-MB
Problems in healthcare can be considered a form of sequential decision making, such as clinical physi- cians making decisions about the best next step
-60
-70
V(n))-WIS
take in care (Ghassemi et al.] 2019). Current advances in Al can provide personalized care that is equal to better than humans by finding optimal decision making policy given clinical data [Ghassemi et al. 2019). Sepsis, a severe infection with organ failure, is leading cause of mortality in intensive care units (ICUs) in hospitals (Sakr et al.) 2018). There are inuuus efforts in research understand and cure sepsis and, hence, increase the survival rates for ICU paticnts. RL for sepsis treat- ment is preferred over supervised learning because the ground truth of good treatment strategy is un- clear in the current medical literature (Marik) 2015). Sepsis data is form of offiine data, from which we can learn an offinn policy given no access to an environment. The challenge lies evaluating such policies while learning to determine when is good enough to be deployed. We use the well-known MTMIC IIT data [Johnson et al. 2016) for scpsis treatment. Current litcrature RL for health (Raghu al. 2017a finds it challenging to evaluate policies learned from such data given no agent-environment interaction.
-90
-100
10
Evaluation Iterations
Figure 5: Results of vanilla off-policy evalua- tion on MountainCar-vo: figure shows results of evaluating a policy, optimized oflline with behavioral cloning, with OPE (without bootstrapping). shows that OPE overestimates the true value of T&.
For this problem, given no further data collection, it is better to rely on offline RL methods (rather than online RL) to Iearn a policy from fixed dataset. A recent work (Killian et al. 2020) explores different representation learning techniques to learn a good policy oflline with discrete BCQ (Fujimoto et al. 2019a).. To evaluate their policies, they rely on Weighted Importance Sampling (Mahmood et al.) 2014) to compare the different representation learning techniques. The problem at hand is widely rescarched but, only this previous work (Killian al.] 2020) relied on offline learning with off off-policy evaluation. While relying on importance sampling for evaluation is good when comparing different approaches, we believe it is essential
11

---

Published in Transactions on Machine Learning Research (9/2024)
to perform evaluation with confidence for this critical application to avoid the risk of overestimation. We build upon their work to apply our proposed training-evaluation framework.
6.2 Experimental Design
We would like to perform conservative evaluation for offline policy learning and see if we can trust our RL policy to make decisions in the real-world. To apply CEOPL, there are a few design decisions to make first. These decisions include: how much data, to use for wraiiiggtessting bootstrapping size, confidence level, and choice of algorithm to optimize a policy offline. Given that the available data is fixed, we split the dataset once in the beginning into a Irain set, used to train the ofllie policy, and a Lest set, used for evaluating the policy performance during training. Note that this fixed batch setup is different from what we followed in Section 4 where we assumed a growing datasct. With stratified sampling (Killian et al. 2020). we use 70/30 train/Vest split, which maintains the same proportions of each terminal oulcome (survival or mortality). Data description and experimental details are shown in Appendix C.1
Unlike simulated environments in Scction 4 we do not have a way in this sctup to evaluate how a policy performs except for using off off-policy estimators. With CEOPL, we use weighted importance sampling (WIS); model-based estimator (MB), and weighted doubly-robust estimator (WDR) with bootstrapping to provide lower-bound estimates eis (Tg) for æy.. We also report ((TA) using the same estimators without bootstrapping, as the OPE estimate. Data is split between training and evaluation. Training data is used to optimize a BCQ agent while testing data is used for evaluation with conservative OPF methods With evaluation, we check if the agent is good enough to deploy in the real-world, and if not, we need to continue improving our oflline agent before deploying. For this problem, we hypothesize that importance sampling (Precup el al., 2000) will provide good estimates given the short horizon and the offine policy constraint method used for improvement, unlike the MB estimator which will find transition dynamics for this problem challenging.
6.3 Results
For sepsis data, there is no ground-truth to compare against, as opposed to the simulated environments we presented in Section E While learning our offline policy, we would like to see how well it performs over time and when it outperforms the behavior policy so we can deploy it into the real world.
Figure f shows the results of applying CEOPL to train and evaluate an offline RL policy, as an average of 10 runs with the standard error indicated. The value of the behavior policy, o(za), is the average return of the data in hand. With WIS, the conservative estimate (s(TT) and the mean estimate ((TO) show the improvement of the policy over training iterations. The mean OPE estimate of WIS slightly outperforms the behavior policy but this can be an overestimate of the true value. We focus on our conservative estimate, which did not show that the target policy outperforms the behavior policy. The conservative estimates by the model-based estimator (MB) failed to detect the improvement. of the poliey over time. This has to do with failing to model the complex dynamics of the dataset; a result, it failed to estimate the true value of the target policy. In a model-based estimator, we model all the environment dynamics, the next state, reward, and terminal stale. In MountainCar-vv0, we only modeled the next state since the reward and terminal state can be computed deterministically from the current state. In Pendulum-vO, the environment always terminates at the maximum horizon, so we only modeled the next state and reward. With sepsis data, was hard enough to model the next state because the state dimension is large; it is also hard to model the terminal state indicating when a patient dies and whether they survive or not.
With weighted doubly-robust estimator, values of (TO) and 6(Ta) can show the improvement of the offline policy as it is training, until both estimates outperform (TO). WDR combines per-decision weighted importance sampling with an approximate model; the approximate model only fits q-values so it does not suffer from the modeling complexity as the MB estimator. However, we cannot roly on the mean estimates ((T)) as they may inaeest For real-world sepsis data, we suggest that relying on OPE only is not enough for such critical application, and it is better to rely on CEOPL for conservative bound estimates
Forr this case study, we face the multi-testing problem in this experiment as we used the same data split for each evaluation iteration; this is because the size of the test. data too small to split among iterations.
12

---

Published in Transactions on Machine Learning Research (9/2024)
Vital
Vm))
430
077
3.5
332
Vim)
0.2
Whol
-322
9afmal
-15
422
-075
-..3
(e(Ra)
-102
-110
1011
Paslustion
Evaliation Fheations
(a) Importance Sampling (WIS)
(b) Direct (MB)
(c) Hybrid (WDR)
Figure 6: Results of CEOPL on sepsis data: this figure shows results of evaluating the performance of the offlie policy. We show both the conservative off- off-polic estimate d/ (T)), and mean OPE estimate i(Te).
because it will be less likely to overestimate. As for the choice of an evaluation method, WIS can be reliable for this problem given the short horizon but WDR can provide better estimates as a hybrid method that improves upon importance sampling. is challenging for the MB estimator detect an improved policy given the complex environment, dynamics.
7 Conclusion
This paper proposed framework for evaluating offline RL methods in a conservative manner (CEOPL), combining OPE with bootstrap confidence intervals. First, we studied the feasibility of different categories of conservative off-policy estimators and how they are affected by offline learning algorithms. We suggest that conservative evaluation is better for safety-critical applications since off-policy evaluation may suffer from overestimation. We tested CEOPL on real-world medical data for sepsis treat While dynamically receiving data, we optimize offline RL agents and run safety tests estimate a lower-bound on the value of the offline policy and control the risk of overestimating its true value. This is essential for safety-critical applications to be able to tell when it is safe enough to deploy new policy. In conclusion, conservative evaluation with CEOPL. proved to be a reliable evaluation method for offline agents that do not have access to a simulated environment. When combined with bootstrapping, direct and hybrid OPE methods can be trusted for evaluating offline agents. The horizon of the trajectory and the environment dynamics affect the choice for the off-policy estimator to rely on. In future work, we plan to tackle learning offline from multiple data sources with different qualities and explore how the quality of the data affects offline learning and conservative evaluation. In addition, we can explore other OPE methods such fitted Q-evaluation (Le) al.) 2019) to eliminate the challenges of direct model-based evaluation, and explore recent approaches for conservative evaluation such as HAMBO (Rothfuss et al. 2023 which are more reliable in continuous state-action spaces.
Broader Impact Statement
We do foresce any potential negative impacts of this rescarch that users need to be aware of. Additionally, our framework provides safe way to perform evaluation of offline RL methods. We have showcased a real- world application of our proposed framework in healthcare, where offline RL can be utilized safely, and hence have a positive impact.
Acknowledgments
Part of this work has taken place in the Intelligent Robot Learning (IRL) Lab at the University of Alberta, which supported in part by research grants from the Alberta Machine Intelligence Institute (Amii); a Canada CIFAR AT Chair, Amii; Digital Research Alliance of Canada; Huawei; Mitacs; and NSERC.
13

---

Published in Transactions on Machine Learning Research (9/2024)
References
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline rein- forcement learning. In International Conference on Machine Learning, pp. 104-114. PMLR, 2020.
Michael Bain and Claude Sammut. A framework for behavioural cloning. In Machine Intelligence 15, Intelligent Agents [St. Catherine's College, Oxford, July 1995), pp. 103-129, GBR, 1999. Oxford University.
ISBN 0198538677.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint or Xrxi:: 01540, 2016.
Carpenter and Bithell. Bootstrap confidence intervals: when, which, what? a practical guide for medical statisticians. Stat Med, 19(9):1141-1164, May 2000.
B. Efron. Bootstrap Methods: Another Look at the Jackknife. The Annals of Statistics, 7(1):1 26, 1979.
doi: 10.
URL https://doi org/10. 1ttp//1...11///////// 1211/22///////34///11
Justin Fu, Mohammad Norouzi, Ofir Nachum, George Tucker, Ziyu Wang, Alexander Novikov, Mengjiao Yang, Michael Rt Zhang, Yutian Chen, Aviral Kumar, Cosmin Paduraru, Sergey Levine, and Thomas Paine. Benchmarks for deep off-policy evaluation. In International Conference on Learning Representations, 2021.
URL /ttpps///penrrevee
Scott Fujimoto, Edoardo Conti, Mohammad Ghavamzadeh, and Joelle Pineau. Benchmarking batch deep reinforcement learning algorithms. arXiv preprint ar 1rxiv:9100 708, 2019a.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pp. 2052 2062. PMLR, 2019b.
Marzych Ghassemi, Tristan Naumann, Peter Schulam, Andrew Beam, Irene Chen, and Rajesh Ranganath. Practical guidance on artificial intelligence for health-care data. The Lancet Digital Health, 1:e157 e159, 08 2019. doi: 1..1016/S2589) 052559977001199
Omer Gottesman, Fredrik Johansson, Matthieu Komorowski, Aldo Faisal, David Sontag, Finale Doshi- and Leo Celi. Guidelines for reinforcement learning in healthcare. Nature Medicine, 25:16-18, 18, 2019.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-poliey maximum entropy deep reinforcement learning with stochastic actor. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Ma- chine Learning Research, pp. 1861-1870. PMLR, 10-15 Jul 2018. URL http://proocedinng mlr .rrrs// #80/haarnojal8b.. html
Josiah P. Hanna, Peter Stone, and Scott Niekum. Bootstrapping with models: Confidence intervals for off- policy evaluation. In Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems, AAMAS '17, pp. 538 546, Richland, SC, 2017. International Foundation for Autonomous Agents and Multiagent Systems.
Josiah P Hanna, Scott Nickum, and Peter Stone. Importance sampling in reinforcement learning with an estimated behavior policy. Machine Learning, 11066):1267-1317, 2021.
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-lanrning In Proceedings of the Thirtieth AAAT Conference on Artificial Intelligence, AAAF1,, pp. 2094-2100. AAAT Press, 2016.
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Àgata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind W. Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. CoRR, abs/1907. 1hs//199700045, 2019. URL
Nan Jiang and Lihong Li. Doubly robust off-policy value evaluation for reinforcement learning. In Interna- tional Conference on Machine Learning, pp. 652 661. PMLR, 2016.
14

---

Published in Transactions on Machine Learning Research (9/2024)
Alistair Johnson, Tom Pollard, Lu Shen, Li-wei Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Celi, and Roger Mark. Mimic-iii, a freely accessible critical care database. Scientific Data, 3:160035, 05 2016. doi: 10.1038/sdata.2115
Taylor W Killian, Haoran Zhang, Jayakumar Subramanian, Mehdi Fatemi, and Marzych Ghassemi. An empirical study of represcntation learning for reinforcement learning in healthcare. In Machine Learning for Health, pp. 139 160. PMLR, 2020.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint drXiv:1312.6114, 2013.
Matthieu Komorowski, Leo Celi, Omar Badawi, Anthony Gordon, and Aldo Faisal. The artificial intelligence clinician learns optimal treatment strategies for sepsis in intensive care. Nature Medicine, 24, 11 2018. doi: 10.1088//415911-18-000233
Ksenia Konyushova, Yutian Chen, Thomas Paine, Caglar Gulcchre, Cosmin Paduraru, Daniel J Mankowitz, Misha Denil, and Nando de Freitas. Active ofliie policy selection. Advances in Neural Information Processing Systems, 34:24631-24665 2021.
Ilya Kostrikov and Ofir Nachum. Statistical bootstrapping for uncertainty estimation in off- off-policy evaluation, 2020.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019.
Aviral Kumar, Anikait Singh, Stephen Tian, Chelsea Finn, and Sergey Levine. A workflow for offline model-free robotic reinforcement learning. In 5th Annual Conference on Robot Learning, 2021. URL
Hoang Le, Cameron Voloshin, and Yisong Yue. Batch policy learning under constraints. In International Conference on Machine Learning, pp. 3703-3712, -3712. PMLR, 2019.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offine reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.011 01643,, 2020.
Yao Liu, Pierre-Luc Bacon, and Emma. Brunskill. Understanding the curse of horizon off-policy evaluation via conditional importance sampling, In International Conference on Machine Learning, pp. 6184 6193. PMLR, 2020.
A. Rupam Mahmood, Hado P van Hasselt, and Richard S Sutton. Weighted importance sampling for off-policy learning with linear function approximation. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems, vol- une 27. Curran Associates, Inc., 2014. URL
05536661004322224477666777868888 Paper. Sraaerpppfff...
P Marik. The demise of early goal-directed therapy for severe sepsis and septic shock. Acta Anaesthesiol Scand, 5955:6611-567, February 2015.
Alanna C. Morrison, Lance A. Bare, Lloyd E. Chambless, Stephen G. Ellis, Mary Malloy, John P. Kane, James S. Pankow, James J. Devlin, James T. Willerson, and Eric Boerwinkle. Prediction of Coronary Heart Disease Risk using a Genetic Risk Score: The Atherosclerosis Risk in Communities Study. American Journal of Epidemiology, 166(1):28-33) 35, 04 2007. ISSN 0002-9262. doi: 10.1093/aje/kwmin https:://oi.coo wtt.://rrr/011.1.1//....
URL
10. 10930.....eeeemmme//
Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. Dualdice: Behavior-agnostic estimation of dis- counted stationary distribution corrections. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, Fox, and R. Garnett (cds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019a. URL https://procednngs. www.:ppscccpppperfiieee...e Fll2cfacccaott511fe8804416600--0
15

---

Published in Transactions on Machine Learning Research (9/2024)
Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. Algaedice: Policy gradient from arbitrary experience, 2019b.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. 2020. doi: 10.46550AARXI222006..00
URL https://arxxv..rrg /bbs/
2006. 09359
Tom Le Paine, Cosmin Paduraru, Andrea Michi, Caglar Gulcehre, Konrad Zolna, Alexander Novikov, Ziyu Wang, and Nando de Freitas. Hyperparameter selection for offline reinforcement learning. arXiv preprint arXi::2007.09055, 2020.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weightel regression: Simple and scalable off-policy reinforcement learning. CoRR, abs/ 1bs/1910.00117,, 2019. URL
1910.00177
Doina Precup, Richard S. Sutton, and Satinder P. Singh. Eligibility traces for off-policy policy evaluation. In Proceedings of the Seventeenth International Conference on Machine Learning, ICML '00, pp. 759 766, San Francisco, CA, USA, 2000. Morgan Kaufmann Publishers Inc. ISBN 1558607072.
Aniruddh Raghu, Matthieu Komorowski, Imran Ahmed, Leo Celi, Peter Szolovits, and Marzych Ghassemi. Deep reinforcement learning for sepsis treatment. arXiv preprint Xrv: 1at:::711.0900. 1.09602. 2017a.
Aniruddh Raghu, Matthieu Komorowski, Leo Anthony Celi, Peter Szolovits, and Marzyeh Ghassemi. Contin- uous state-space models for optimal sepsis treatment: deep reinforcement learning approach. In Machine Learning for Healthcare Conference, pp. 147-163. PMLR, 2017b.
Martin Riedmiller. Neural fitted q iteration first experiences with a data efficient neural reinforcement learning method. In João Gama, Rui Camacho, Pavel B. Brazdil, Alípio Mário Jorge, and Luís Torgo (eds.), Machine Learning: ECML 2005, pp. 328, Berlin, Heidelberg, 2005. Springer Berlin Heidelberg. ISBN 978-3-540-31692--3.
Jonas Rothfuss, Bhavya Sukhija, Tobias Birchler, Parnian Kassraie, and Andreas Krause. Hallucinated adversarial control for conscrvative offline policy evaluation. In Robin J. Evans and Πlya Shpitser (eds.), Proceedings of the Thitty-Ninth Conference on Uncertainty in Artificial Intelligence, volume 216 of Proceedings of Machine Learning Research, pp. 1774-1784. PMTR, 31 Jul-04 Aug 2023. URL
https://procceedings mlr. wwwpppewweee...eeeep..pp..e...ae h2tps////2//rroos222222 html
Yasser Sakr, Ulrich Jaschinski, Xavier Wittebole, Tamas Szakmany, Jeffrey Lipman, Silvio A Namendys Silva, Ignacio Martin-Loechos, Mare Leone, Mary-Nicoleta Lupu, Jean-Louis Vincent, and ICON Inves- tigators. Sepsis in Intensive Care Unit Patients: Worldwide Data From the Intensive Care over Nations Audit. Open Forum Infectious Diseases, 5(12), 11 2018. ISSN 2328-8957. doi: 10. off3//o3.../..33.133. URL https://dot 10.g/10.1000 wrtg//..193//f1.../oooo 109.1000f/f////f....... ofy313.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018.
Philip Thomas and Emma Brunskill. Data-officient off-policy policy ovaluation for reinforcement learning. In International Conference on Machine Learning, pp.
PMLR, 2016.
Philip Thomas, Georgios Theocharous, and Mohammad Ghavanzadeh. High-confidence off-policy eval- uation. Proceedings of the AAAI Conference on Artificial Intelligence, 29(1). Feb. 2015a. doi: 10.1609/amm v29il
URL https:://js hww.lloo//inddee pptppp/AA//tttccee///eee/ee////e
Philip Thomas, Georgios Theocharous, and Mohammad Ghavamzadeh. High confidence cocieleneepolyyyyyoeee improvement. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learnång, volume 37 of Proceedings of Machine Learning Research, pp. 2380-2385 Lille, France, 07-09 Jul 2015b. PMLR. URL httts:://proceedingg mlr, .tt....ees/v3//// 1tt//3//////11.. thomas15.... 1..mml
Philip S Thomas Safe reinforcement learning. PhD thesis, University of Massachusetts Libraries, 2015.
16

---

Published in Transactions on Machine Learning Research (9/2024)
Cameron Voloshin, Hoang M Le, Nan Jiang, and Yisong Yuc. Empirical study of off- policy evaluation for reinforcement, learning. arXiv preprint arXiv: 1.t:1911.00844 2019.
Christopher J. C. H. Watkins and Peter Dayan. Q-learning. In Machine Learning, pp. 279-292 1992.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized ofline reinforcement learning. arxiv
preprinl Xiv:1911 111361, 2019.
Mengjiao Yang, Ofir Nachum, Bo Dai, Lihong Li, and Dale Schuurmans. Off- Off-policy evaluation via the regularized lagrangian. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Newral Information Processing Systems, volume 33, pp. 6551 6561. Cur- ran Associales, Inc., 2020. URL BB8-:104420daa12728cclllllaaaaaa
neurips neurips.cc/paper files//papprr/222//illl/
pdf
Shangtong Zhang, Bo Liu, and Shimon Whiteson. GradientDICF: Rethinking generalized offline estimation of stationary values. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 11194 11203. PMLR, 13-18 Jul 2020. URL https::/proceeeeeee
mlr, press/v119/ 22a.gg00 html
17

---

Published in Transactions on Machine Learning Research (9/2024)
A Appendix: Methodology
This section provides further details of the methods used in CEOPL.
A.1 Offline Reinforcement Learning
As mentioned in Section 3.1 we categorized learning methods into three categories: normal off-policy RL algorithms (off-policy), imitation learning tochniques (imitation learning), and policy constraint methods specific for offline learning (offine).. For continuous control, we describe the approaches in the three cate- gories. For the off-policy category, we used soft actor-critic (SAC) (Haarnoja et al. 2018); this means a SAC agent is optimized offline using a fixed buffer of data without interaction with the environment or control over the buffer. For the imitation learning category, we study Behavioral Cloning (BC) (Bain & Sammut, 1999); BC is a supervised learning algorithm where the model learns how to predict actions from the current state.
In the offiine category, we study batch- conetrannst Q-learning (BCQ) (Fujimoto et al. 2019b and bootstrap- ping error accumulation reduction (BEAR) (Kumar et al. 2019). contta-cnst Q-learning (Fujimoto ct al., 2019b) argue that off-policy algorithms often fail the offline sctting due to the extrapolation error since unseen stale-action pairs will have unrealistic values (Fujimolo el. al., 2019b). This is a result of the mismatch between the state-action visitation of the current policy and the state-action pairs in the offline dataset. To solve this, the policy should induce a similar state-action visitation to the offline data. BCQ FFujimoto et al. 2019b) proposed the use of state-conditioned generative model to produce only likely ac- tions to the current offline data. Then, this generative model, effectively a variational auto-encoder (Kingma & Welling 2013), is combined with a network which aims to optimally perturb the generated actions in a small range. When this combined with the Q-network, the network will only select the highest valued actions similar to the data in the batch (Fujimoto al. 2019b).
BEAR (Kumar et al., 2019) argues that the source of instability for off-policy algorithms learning offline is the bootstrapping error. This error results from boolstrapping with actions that lie outside of the training data distribution, and it accumulates via the bellman backup operator (Kunar et al.) 2019]. BEAR is built on top of any actor-critic algorithm, such as SAC (Haarnoja ct al.) 2018), by modifying the policy improvement step to use a distributtion-ccoslrriiin
backup. To apply the constraint, the sampled version of maximum-nee -ean mean discrepancy is used between the unknown behavior policy ny and the current actor To constrain the distribution of the actor to the support of the behavior policy (Kumar et al., 2019). Hence, BEAR allows the actor to maximize the Q-function while being constrained to remain in the valid support space of the behavior policy defined by the data samples [Kumar et al. 2019).
For discrete control, we use Double DQN (Hasselt et al. 2016) as a normal off-policy RL algorithm; similar to SAC, Double DQN is used to learn solely from offline data. For the imitation learning category, we use behavioral cloning (BC) (Watkins & Dayan 1992) in a supervised learning manner with cross-entropy loss. In the offline category, we experiment with the discrete version of BCQ {Fujimoto al. 2019a). Discrete BCQ is much simpler than its continuous version, since it trains Q-learning with constrained argmaax operator. This only allows actions in the backup with probability, given by the generative model, above some threshold т. The generative model is a behavioral cloning network trained in standard supervised learning with a cross-entropy loss (Fujimoto et al. 2019a). BCQ is also based on Double DQN, and we use the threshold to be x = 0.3 as reported in their original paper. If the threshold T is 1, the algorithm returns an imitator of all the actions in the dataset, while a threshold T = O returns the Q-learning objective.
A.2 Off-Policy Evaluation
Importance Sampling (rrecup et al. 2000 is a method for handling mismatch between distributions and hence presented as a consistent and unbiased off-policy estimator. For trajectory H ~ mh of length L, as H 81, (1, Fl my 81, 0t, PL, we can define the importance sampling up to time for policy ma as follows: rI,
m
To (A
IS To. D) -)= Σ
pt HR. P
H
(1)
m 52)
i=1
18

---

Published in Transactions on Machine Learning Research (9/2024)
By re-weighting the returns, can tell how likely each reward is under we versus Tb. However, IS often assurees a known Th, which is not always the case for off-policy policy evaluation. Since assume no access to Tb in our setting, we estimate a behavior policy 4t. from data with a behavior cloning model (Hanna et al, 2021j. To come up with a high confidence estimate, we first compute the importancc-weightee returns then use bootstrapping, described in Section 3.2 to get. the lower-bound estimate. In our experiments, we use weighted importance sampling (WIS) (Mahmood et al. 2014p with bootstrapping as a representation for importance sampling family of estimators.
P g(Hi) 2==1 PL-1
WIS( Db
(2)
Direct Model-Based Estimator another off-policy estimator that falls under direct methods. The model-based off-policy estimator (MB) computes ((m)) by building a model using all the available trajectories D to build a model M (S, A, P. r,,, do) where P and dy are estimated with trajectories sampled from Tg. Then, MB will compute (TT) as the average return trajectories simulated in the estimdted model M while following iTg. Despite having lower variance than IS methods, MB biased estimator for: 1) lack data for particular state-action pairs, so assume their transition probabilities, 2) assume that the model class includes the true model for the transitions. As result, as oc, the model estimates may converge to value different from the true v(To). This effect is duc to its dependency on the modeling assumptions we make whether we assume a linear or non-linear model.
Therefore, the model- based estimator is combined with bootstrapping, as discussed in Section 3.2 for con- servative off-policy evaluation. To clarify, as mentioned in Section A.1 we rely mainly on model-free policy optimization algorithms for learning policy offline. However, we model-based estimation evaluate policy, which is independent from the policy optimization part.
Weighted Doubly-Robust Estimator TTomas & Brunskill 2016 is a hybrid method for off-policy ostimation, presented as an extension to the doubly-robust (DR) mothod (Jiang & Li 2016). DR is an unbiased estimator of (TT)) that uses an approximate model of the MDP to reduce the variance of importance sampling (Jiang & Li 2016). Although biased, WDR is based per-decision weighted importance sampling (PDWIS) and improves upon the DR method as it balances the bias-variance trade-off. In addition, the approximate model value functions act as control variate for PDWIS.
L-1
P
PDWIS( D,mb)
R
(3)
j== P
WDR( D, PDWIS( D,
(4)
* (wiana (S), At) w(-10pio(S))
Similar to direct model-based methods, we use bootstrapping with WDR to provide confidence lower- bound estimate on Ú(Te). WDR with bootstrapping is guaranteed to converge to the correct estimate as n increases, given the statistical consistency of PDWIS (Hanna et al.] 2017). For the approximate model, a single model estimated with the available trajectories D, then used to compute the value functions of WDR for each bootstrap data. We choose the weighted doubly robust. estimator to represent OPE hybrid methods in our study.
A.3 Bootstrapping
The pseudo-code of Bootstrap Confidence Intervals (BCT) is shown in Algorithm
19

---

Published in Transactions on Machine Learning Research (9/2024)
Algorithm 2 Bootstrap Confidence Intervals: BCI
Input: a target policy næ, dataset Dtest of m trajectorics, confidence level 8 E [0, 1], number of bootstrap estimates B, an estimate of the behavior policy di. and off-policy estimator Yp Output: (TT): 1 lower-bound i(To)
1: for j [1 B) do
2: D D Hi Hi... Hy where Hi is sampled uniformly
V/ (To, Dj, m6)
end for
sort(eyyl j E [1, BJ) // ascending
1= [
7: return dy
B
Appendix: Empirical Results
This section provides further details the experimental setup and further analysis of the empirical results the simulated environments.
B.1 Simulated Environments
Car-tainCar-vO: we use discrete MountainCar (Sutton & Barto 2018) with a continuous state (velocity and position) and 3 possible discrete actions. At each time-step, the reward is -1, except for the terminal state when it is 0. However, we used the modified version of Mountain-Cas as described here (Thomas 2015). The horizon is shortened by holding an action a₃ constant for 4 updates of the environment state. We also change the start state such that an episode starts with a random position in the range of (-1.2, 0.6) and random velocity in the range of (-0.07, 0.07) (Jiang & Li 2016 Thomas 2015). The data collector we use for this environment is an online actor-critic (Sutton & Barto 2018) agent that is partially trained with added 30% randomization when generating data. This means when collecting the offline data, an agent takes a random action 30% of the lime instead of following the online policy, to include exploratory transitions
Pendulum-v0:
inverted pendulum swing-up problem a classic problem the control literature with one continuous action (Brockman al., 2016). In this version of the problem, the pendulum starts in a random position, and the goal is to swing it up stay upright. We also shorten the horizon following the same procedure as done in MountainCar [Thomas 2015 by holding action constant for 4 updates of the enviromment state. This limits the horizon of the environment to 50 instead of 200. The data source we use for this environment is soft actor-critic (Haarnoja et al. 2018) agent that is trained partially, with added 30% randomization when generating the dataset. This means when collecting the offline data, an agent takes a random action 30% of the time instead of following the online policy to resemble real-world data.
B.2 Overestimation of CEOPL
Figure 7 shows the overestimation of lower bounds compared to the true value of the offline policy; is guaranteed to remain within the allowable 8 - 5% error rate for 95% confidence, We would like to point out that the safety guarantee for the confidence bound specified is per-iteration (Thomas 2015). If we want to consider this guarantee over multiple iterations, the probability - 5 decreases for the early iterations we perform more tests. As a result, we should only consider this guarantee for the last iteration of our loop, such that the error rate in the last iteration is at most 8.
B.3 Further Analysis
To better understand how evaluation is affected by the offline learning method, multiple measures can show the similarity between two probability distributions (as policy distribution over actions). One simple measure is the total variation (TV) distance. TV distance measures the difference between action probabilities taken under two policies given the data set in hand. TV would be the sum of differences in
20

---

Published in Transactions on Machine Learning Research (9/2024)
K15
MB
WDH
low -o----dddd etrer rate
0.0
0.15
0.10
0.02
0.00
(a) Double DQN
0.30
0.2
lower lower-booee teee--buuddd etrot rate
0.0
10
Evaluation Iterations
(b) Behavior Cloning (BC)
0.50
(owerbound CTtSr obe
615
d 0.05
Eraduation Baralion
(c) Datch-Constrained Q-learning (BCQ)
Figure 7: The empirical error rate on MountainCar-vo with CEOPL: This figure shows the empirical error rate MountainCar given different evaluation methods. The lower bound is computed times for each method (z 40 for Mountain Car) and count how many times the lower bound is above the true i(TT)) computed in the true environment. Given 200 trajectories only for evaluation, all methods correctly approximate the allowable 5% error rate for a 95% confidence lower bound. However, there are two instances of WIS and WDR that slightly exceed 5%, which can be mitigated with more data.
probabilitics between the behavior policy nu and the target policy for cach state-action pair in the test dataset. Since ma is not known, we estimate कं given the test. data (Hanna el. al. 2021). TV distance is correlated to KL-divergence, showing how two policies are different from each other. As discussed Section and shown in Figuro g in MountainCar-vo (discrete control), behavior cloning (BC) learns a policy that is very similar to î.. Discrete BCQ and double DQN do not limit the distance with the behavior policy, and their TV distance is quite large opposed to BC. For Pendulum-v0 (continuous control), behavior cloning
21

---

Published in Transactions on Machine Learning Research (9/2024)
3000
DQN
14000
12000
2500
TV Distance
2000 BCQ
TV Distance
8000
BEAR
6000
1000
BC
2000
6
10
6
Evaluation Iterations
Evaluation Iterations
(a) MountainCar-vo (discrete control)
(b) Pendulum-vO (continuous control)
Figure 8 Divergence between ft. सe over training iterations: this figure shows the total variation (TV) distance between A& & wo MountainCar-v0 & Pendulum-vO over all training iterations. For both discrete (a) and continuous control (b), BC achieves the lowest distance since it tries to mimic the data distribution. In (a), Double DQN and BCQ policies achieve similar distance, but higher compared to BC. (b), BEAR achieves a higher distance than BC, but much less than BCQ; this is because BEAR enforces explicit divergence minimization between d. and nz.
(BC) also learns a policy that very similar to is and achieves the lowest TV distance. BEAR limits the divergence compared to BCQ since BEAR's objective is constrained to be close to Te, as shown in Figure
C
Appendix: Real-world Case Study
This section provides further experimental details and analysis of the experiments on sepsis data.
C.1 Experimental Details
To build a decision decision-making policy for the treatment of septic patients, we use data from the Medical Informa tion Mart for Intonsive Care (MIMIC-III) dataset (v1.4) (Johnson et al. 2016). We follow previous relovant work (Komorowski et al. 2018 to extract and preprocess vital and measurements and build a cohort of 19418 patients; this cohort has an observed mortality rate just above 9% (determined by death within 48h of the final observation). Data extraction, preprocessing, and splitting follow previous work (Killian al. 2020
A dataset D is collection of trajectories; a trajectory H of non-fixed horizon L refers to a single ient that ends by the survival or death of patient, with a maximum horizon L of 19. A transition is composed of current observation Ot, the action taken by the clinician as to move to the next observation Ox-1 receiving reward of r, until a patient survives or dies. We treat the problem as markov decision process (MDP). Data consists of continuous state space of a dimension 38 and a discrete action space of 25 actions. Obscrved actions are the administration of fluids or vasopressors that can be given, categorized by volune, and put into 5 discrete bins per action type resulting in 25 actions (Killian et al. 2020). The data has sparse reward such that trajectory has terminal reward of +1 if patient survived, -1 if a patient died, and 0 otherwise. The observations used as our state space are combination of time-varying continuous features (33), such as heart rate, glucose, etc., and demographic features (5), such as gender, age, weight, etc. (Killian et al. 2020).
For the bootstrapping size, we use 2000 as in the experiments with simulated data. We also use a confidence level å = 0.05. We optimize policy offline using baich--onsttained Q-learning (BCQ) for discrete
22

---

Published in Transactions on Machine Learning Research (9/2024)
control (Fujimoto et al. 2019a), coupled with auto-encoder for better state representation learning. Another important decision is to decide how often we will run our conservative evaluation methods because of its computational cost. For this, we train the policy for 200 epochs, where we evaluate the policy every 10k epochs. To estimate the behavior policy Te, which is referred to as the clinician's policy, we rely on behavior cloning (Hanna et al. 2021 as done other experiments on simulated environments.
23


Reviews:
[{"review_id": "VOMVKgFeXk", "reviewer": "Reviewer_qFaC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n- Evaluation of policies is an important open question, and the paper proposes a sound approach towards solving it. \n\nWeaknesses:\n- The method formulation looks not much different from the standard bootstrapping method, so it would be important to emphasise how this algorithm would provide an additional theoretical or empirical value for the readers comparing to existing RL bootstrapping literature, why cannot we gain the same results from, say, Algorithm 1 from Wang et al (2022)? It seems like the same exact results could be achieved with other bootstrapping algorithms for offline RL? If not the authors need to emphasise how exactly this cannot be solved with the existing methods (hence the current score on the audience criterion). \n- The experimental evaluation only concerns some simple baselines, which limits the experimental contribution; common gym environments such as halfcheetah, hopper, walker2d could be recommended to improve the experimental evaluation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors address the question of evaluation of offline policy under the assumption that neither the environment nor the data-generating policy are known.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "*\"In the offline RL literature, there is no standard way to evaluate a policy while\nlearning offline; most of the literature still evaluates the performance of offline algorithms in the environment\nor in a simulator, by running a policy online.\"*\n\nI think this statement should be put in a wider context. While it is the case that the evaluation of *reported* performance is done in the simulator, numerous works use policy evaluation proxies as a part of the optimisation procedure, including bootstrapping such as in Wang et al (2022) and the cited paper by Hanna et al (2017). There is, however, another related question: can we provide, just based on the offline data, any theoretical guarantees on, for example, the distribution match or any other discrepancy between the offline estimate trajectory and the simulator evaluation? Without such guarantees, it does not answer the original claimed question : \"*how to best learn offline and (confidently) select the right time to deploy, if we do not have access to the environment nor the policy generating the data?*\" \n\nFurther to that, I may just miss the point but it would be good if the authors confirm how the proposed method actually answers 'how to best learn offline'. It appears that the answer to this question is not provided. It is because of these two points that I put the claims and evidence score to 'no' for now. \n\nAlso, experimental analysis needs to be improved as detailed above. \n\nWang, Kerong, et al. \"Bootstrapped transformer for offline reinforcement learning.\" Advances in Neural Information Processing Systems 35 (2022): 34748-34761.", "broader_impact_concerns": "No ethical concerns provided", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ElLxgnZFYQ", "reviewer": "Reviewer_pbpT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The authors provide a good introduction to the OPE literature as well as some of the offline RL literature. The application of bootstrapping to get confidence intervals is a quick but reasonable and effective way to get a lower-bound estimate on the value.\n\nI think the primary weaknesses of the paper are:\n\n1) The methods used are a bit far from SOTA for offline RL. Last I checked I believe BEAR and BCQ were considered subpar, in favor of CQL and TD3+BC.\n\n2) The authors claim a contribution of \"exploring the effect of offline learning on OPE methods given constantly-improving target policy, as opposed to fixed policy\" seems a little hollow. Yes, it is true that many OPE papers present experiment tables for evaluating a fixed policy on a varying number of episodes, but almost every one of those papers in practice is implemented by applying it to a continually learning setup.\n\n3) In general, I found it hard to have a takeaway from the experiment results, aside that bootstrapping led to lower estimates of the offline value function.\n\nI think that the primary contribution of the paper is the application of bootstrapping, except this has been done before in prior work and so I'm not sure this has *any* novel contribution, even a modest one.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors discuss off-policy evaluation for offline learning, and in particular how it would be helpful to have confidence intervals on these estimates such that we could have confident lower bounds. In particular the authors place focus on evaluating the performance of a constantly improving target policy, rather than a fixed checkpoint.\n\nFor the OPE method, the authors consider an importance sampling method, a model-based method, and a hybrid approach a la the common weighted doubly-robust method. For the offline RL algorithm, they ablate BC, BCQ, and BEAR.\n\nTo get confidence intervals, when doing the off policy evaluation, they do bootstrap resampling of the off-policy estimate $v(\\pi_theta)$.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "I would have appreciated more discussions of when bootstrapping was applied in off-policy evaluation. It's certainly not a new idea (see https://proceedings.mlr.press/v139/hao21b.html for example.)", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "N3XnwR2wVh", "reviewer": "Reviewer_Ksxj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The paper is well-written.\nWeaknesses: The paper lacks theoretical results.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a framework for conservative evaluation of offline policy learning (CEOPL). They focus on being conservative so that the probability that the agent performs below a baseline is approximately $\\delta$, where $\\delta$ specifies how much risk we are willing to accept. They assume access to a data stream, split into a training set to learn an offline policy, and a testing set to estimate a lower bound on the offline policy using off-policy evaluation with bootstrap confidence intervals.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. While the authors claim that a lower bound estimate of policy evaluation is established. There is no theory to support that $\\hat v$ produced by Algorithm 1 is a lower bound with probability $1-\\delta$.\n2. I can see the benefit of conservative approach in policy learning. But for policy evaluation, if one $\\pi$ is overestimated, would every $\\pi$ be overestimated? So just be a constant shift?", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers have not reached an argument on this paper, but overall I would lean towards acceptance.

First, one of the negative reviews is justified with a very short "The paper lacks theoretical results.",and I do not think this is enough reason to reject this work. Authors provide sufficient empirical evidence for the usefulness of their framework.

I agree that, from a practitioner's perspective, it does seem new to consider OPE during learning. However, I think we can appreciate that the authors tackle an issue that is strongly relevant to the offline RL community.
Decision: Accept (poster)