Data-Efficient Pipeline for Offline Reinforcement Learning with Limited Data
Authors: Allen Nie
Yannis Flet-Berliac
Deon Richmond Jordan
William Steenbergen
Emma Brunskill
Venue: NeurIPS 2022 Accept (2022)
Keywords: offline reinforcement learning
hyperparameter selection
policy deployment
small data regime
batch value function tournament (BVFT)

Abstract: Offline reinforcement learning (RL) can be used to improve future performance by leveraging historical data. There exist many different algorithms for offline RL, and it is well recognized that these algorithms, and their hyperparameter settings, can lead to decision policies with substantially differing performance. This prompts the need for pipelines that allow practitioners to systematically perform algorithm-hyperparameter selection for their setting. Critically, in most real-world settings, this pipeline must only involve the use of historical data. 
Inspired by statistical model selection methods for supervised learning, we introduce a task- and method-agnostic pipeline for automatically training, comparing, selecting, and deploying the best policy when the provided dataset is limited in size. In particular, our work highlights the importance of performing multiple data splits to produce more reliable algorithm-hyperparameter selection. While this is a common approach in supervised learning, to our knowledge, this has not been discussed in detail in the offline RL setting. We show it can have substantial impacts when the dataset is small. Compared to alternate approaches, our proposed pipeline outputs higher-performing deployed policies from a broad range of offline policy learning algorithms and across various simulation domains in healthcare, education, and robotics. This work contributes toward the development of a general-purpose meta-algorithm for automatic algorithm-hyperparameter selection for offline RL.

Data-Efficient Pipeline for Offline Reinforcement Learning with Limited Data
Allen Nie
Yannis Flet-Berliac
Deon R. Jordan
William Steenbergen
Emma Brunskill
Department of Computer Science
Stanford University aniiesstanfore edu
Abstract
Offline reinforcement learning (RL) can be used to improve future performance by leveraging historical data. There exist many different algorithms for offline RL, and it is well recognized that these algorithms, and their hyperparameter settings, can lead to decision policies with substantially differing performance. This prompts the need for pipelines that allow practitioners to systematically perform algorithm- hyperparameter selection for their setting. Critically, in most real-world settings, this pipeline must only involve the use of historical data. Inspired by statistical model selection methods for supervised learning, we introduce a task- and method- agnostic pipeline for automatically training, comparing, selecting, and deploying the best policy when the provided dataset is limited in size. In particular, our work highlights the importance of performing multiple data splits to produce more selection. While this is common approach in supervised learning. to our knowledge, this has not been discussed in detail in the offline RL setting. We show can have substantial impacts when the dataset is small. Compared to alternate approaches, our proposed pipeline outputs higher-perfone perforrner deployed policies from broad range of offline policy learning algorithms and across various simulation domains in healthcare, education, and roboties. This work contributes toward the development of general-purposs meta-algorithm for automatic Soorithm--hperppraame selection for offline RL.
reliable
Introduction
Offline/batch reinforcement learning has the potential to learn better decision policies from existing real-world datasets on sequences of decisions made and their outcomes. many of these settings, tuning methods online is infeasible and deploying new policy involves time, effort and potential negative impact. Many of the existing datasets for applications that may benefit from offline RL may be fairly small comparison to supervised machine learning. For instance, the MIMIC intensive care unit dataset sepsis that is often studied in offline RL has 14k patients (Komorowski et al., 201 8), number of students frequently interacting with an online course will often range from hundreds to tens of thousands (Bassen et al., 2020), and the number of demonstrations collected from a human operator manipulating robotic arm often on the order of few hundred per task (Mandlekar et al., 2018). In these small data regimes, recent studies (Mandlekar et al., 2021: Levine et al., 2020) highlight that with limited data, the selection of hyperparameters using the training set is often challenging. Yet hyperparameter selection also has a substantial influence on the resulting policy's performance, particularly when the algorithm leverages deep neural networks.
One popular approach address this learn policies from particular algorithm-hypeet hyperaaaatttrrrrtttee pairs on a training set and then use offline policy selection, which selects the best policy given validation set (Thomas et al., 2015a, 2019; Paine et al., 2020; Kumar et al., 2021). However, when
36th Conference on Neural Information Processing Systems (NeurIPS 2022).

---

Data Efficient (re-train)
Compare Across OPL
Considers
Considers
Non-Markov Env
Common Practices
Evaluation Variation
Training
Variation
Policy selection (1 split)
Internal Objective / TD-Error
(depends)
x
x
x
(Thomas et al., 2015b, 2019)
OPE methods
(Komorowski et al. (2018); Paine al. (2020)
(depends)
OPE BCa Val. (Thomas al., 2015b)
(depends)
X
X
BVFT
x
X
X
X
(Xie and Jiang, 2021)
BVFT OPE
x
X
/
X
(Zhang and Jiang. 2021)
Q-Function Workflow
x
X
(Kumar et al., 2021)
Ours: A. selection (multi-split)
Volss-Valiattion
/
Repeated Random
Subsampling
Table 1: A summary of commonly used approaches for choosing deployment policy from fixed offline RL dataset. We define Data Efficient as: the approach assumes the algorithm can be re-trained on all data points; (depends) as: depends on whether the underlying OPL or OPE methods make explicit Markov assumption or not.
the dataset is limited in size, this approach can be limited: (a) if the validation set happens to have no or very few good/tigh-reward trajectories, then trained policies cannot be properly evaluated; (b) if the training set has very few such trajectories, then no good policy behavior can be learned through any policy learning algorithm; and (c) using one fixed training dataset prone to overfitting the hyperparameters on this one dataset and different hyperparameters could be picked the training set changes. One natural solution this problem to train on the entire dataset and compare policy performance on the same dataset, which is often referred to as the internal objective approach. In Appendix A. conduct short experiment using D4RL where this approach fails due to the common issue of Q-value over-estimation (Fujimoto et al., 2019).
There has been much recent interest in providing more robust methods for offline RL. Many rely on the workflow just discussed, where methods are trained on one dataset and Offline Policy Evaluation (OPE) is used to do policy selection (Su et al., 2020; Paine et al., 2020; Zhang and Jiang, 2021; Kumar al., 2021; Lee et al., 2021; Tang and Wiens, 2021; Miyaguchi, 2022). Our work highlights the impact a less studied issue: the challenge caused by data partitioning variance. We first motivate the need to account for train/validation partition randomness by showing the wide distribution of OPE scores the same policy can obtain on different subsets of data the very different performing policies the same algorithm and hyperparameters can learn depending on different training set partitions. We also prove single partition can have notable failure rate in identifying the best algorithm--hyperprameee to learn the best policy.
(AH) selection and policy de- We then introduce a general pipeline for sub-samp sampling (RRS) with replacement of the dataset to ployment that: (a) uses repeated random sub-
perform AH training, (b) uses OPE on the validation set, (c) computes aggregate statisties over the RRS splits to inform AH selection, and (d) allows to the selected AH to retrain the entire dataset to obtain the deployment policy. Though such repeated splitting is common in supervised learning, its show
impact and effect have been little studied in the offline RL framework. Perhaps surprisingly, that our simple pipeline leads to substantial performance improvements a wide range of popular benchmark tasks, including D4RL (Fu et al., 2020) and Robomimic (Mandlekar et al., 2021).

---

2
Related work
Offline Policy Learning (OPL), In OPL, the goal is to use historical data from a fixed behavior policy Tr, to learn a reward-maxinizing policy in an unknown environment (Markov Decision Process, defined in Section 3). Most work studying the sampling complexity and efficiency of offline RL (Xie and Jiang, 2021; Yin et al., 2021) do not depend on the structure of a particular problem. but empirical performance may vary with some pathological models that are not necessarily Markovian. Shi et al. (2020) have precisely developed a model selection procedure for testing the Markovian hypothesis and help explain different performance on different models and MDPs. To address this problem, is inherently important to have fully adaptive characterization in RL because it could save considerable time in designing domain-speciffe RL solutions (Zanette and Brunskill, 2019). As an answer to a variety of problems, OPI. is rich with many different methods ranging from policy gradient (Liu et al., 2019), model-based (Yu et al., 2020; Kidambi et al., 2020), to model-free methods (Siegel et al., 2020; Fujimoto et al., 2019; Guo et al., 2020; Kumar et al., 2020) each based on different assumptions on the system dynamics. Practitioners thus dispose of an array of algorithms and corresponding hyperpa- rameters with no clear consensus on a generally applicable evaluation tool for offline policy selection.
Offiine Policy Evaluation (OPE). OPE is concerned with evaluating target policy's 'sicy's performance using only pre-collected historical data generated by other (behavior) policies (Voloshin et al., 2021). Each of the many OPE estimators has its unique properties, and in this work, we primarily consider two main variants (Voloshin et al., 2021): Weighted Importance Sampling (WIS) (Precup, 2000) and Fitted Q- Q-Evaluation (FQE) (Le et al., 2019). Both WIS and FQE are sensitive to the partitioning of the evaluation dataset. WIS is undefined on trajectories where the target policy does not overlap with the behavior policy and self-normalizes with respect to other trajectories in the dataset. FQE learns a Q-function using the evaluation dataset. This makes these estimators very different from mean-squares errors or accuracy in the supervised learning setting - the choice of partitioning will first affect the function approximation in the estimator and then cascade down to the scores they produce.
Offline Policy Selection (OPS). Typically, OPS is approached via OPE, which estimates the expected return of candidate policies. Zhang and Jiang (2021) address how to improve policy selection in the offline RL setting. The algorithm builds on the Batch Value-Function Tournament (BVFT) (Xie and Jiang, 2021) approach to estimating the best value function among set of candidates using piece-wise linear value function approximations and selecting the policy with the smallest projected Bellman error in that space. Previous work on estimator selection for the design of OPE methods include Su et al. (2020); Miyaguchi (2022) while Kumar et al. (2021); Lee et al. (2021); Tang and Wiens (2021); I); Paine et al. (2020) focus on offline hyperparameter tuning. Kumar et al. (2021) give recommendations on when to stop training model to avoid overfitting. The approach is exclusively designed for Q-learning methods with direct access to the internal O-functions. On the contrary. our pipeline does policy training, selection, and deployment on any offline RL method, not reliant on the Markov assumption, and can select the best policy with potentially no access the internal approximation functions (black box). We give a brief overview of some OPS approaches in Table 1.
3 Background and Problem Setting
We define a stochastic Decision Process M M= = A, A, T.r.). 7), where S is set of states; A is set of actions; T is the transition dynamics (which might depend on the full history); r is the reward function; and 7 (0, 1) is the discount factor. Let 7 {ss, a, s/ 1/) be the trajectory sampled from ग on M. The optimal policy π is the one that maximizes the expected discounted return ((x)=E,<p))) [G[T)) where G(r)= Di=o I'r: and p= is the distribution of under policy π. For simplicity, in this paper we assume policies are Markov π + A, but straightforward to consider policies that are function of the full history. In an offline RL problem, we take a dataset: group of policies which
D = {r. /=1 which can be collected by one policy ne on the decision process M. The goal in offline/batch RL is to learn a decision policy x from class of policies with the best expected performance V for future use. Let A, to denote an AH pair, Le. an offfine policy learning algorithm and its hyperparameters and model architecture. An offline policy estimator takes in policy ne and a dataset D, and returns an estimate of its performance: V V x D > R. In this work, we focus two popular Offline Policy Evaluation (OPE) estimators: Importance Sampling (IS) (Precup, 2000) and Fitted Q-Evlluation (FQE) (Le et al., 2019) estimators. We refer the reader to Voloshin et al. (2021) for a more comprehensive discussion.
refer to the hehavior
3

---

Yoe 5owarde5.36
You 1.04
MBP
PHOP AH--5
Polcs
Tue 4.11
MBS AH 23-
Tue Rewardd-44.6 4.062
Nesgl
MBS 0 AH 13-
Reward 5.06
Truc Roward- 13.25
BC A21-6
30
-20
-10
30
50
-20
-15
OPE Estimates on 10 Partitions
True Reward of Policy trained on 10 Partitions
(b)
Figure 1: True performance and evaluation of 6 A; pairs on the Sepsis-POMDI (N=1000) domain. (a)
shows the OPE estimations and (b) shows the variation in terms of true performance. The variations are due to the different AH pairs of the policies but also to the sensitivity to the training/validation splits.
4 The Challenge of Offline RL A, Selection
An interesting use-case of offline RL is when domain experts have access to existing dataset (with potentially only a few hundred trajectories) about sequences of decisions made and respective outcomes, with the hope of leveraging the dataset to learn a better decision policy for future use. In this setting, the user may want to consider many options regarding the type of RL algorithm (model-based model-free, or direct policy search), hyperparameter, or deep network architecture to use.
Automated algorithm selection is important because different A, (different AH pairs) may learn very diverse policies, each with significantly different performance VAA, Naturally, one can expect that various algorithms lead to diverse performance, but using a case-study experiment on a sepsis simulator (Oberst and Sontag, 2019), we observe in Figure 1(b) that the sensitivity to hyperparameter selection is also substantial (ef. different average values in box plots for each method). For example, MBS-QI (Liu et al., 2020) learns policies ranging from over -12 to -3 in their performance, depending on the hyperparameters chosen.
Precisely. to address hyperparameter tuning, past work often relies on executing the learned policies in the simulator/real environment. When this is not feasible, as in many real-world applications, including sepsis dataset example, where the user may only be able to leverage existing historical data, we have no choice but to rely on off-policy evaluation. Prior work (Thomas et al., 2015b; Farajtabar et al., 2018; Thomas et al., 2019; Mandlckar et al., 202 1) have suggested doing so using hold-out method, after partitioning the dataset into training and validation sets.
Unfortunately, the partitioning of the dataset itself may result in substantial variability in the training process (Dietterich, 1998). We note that this problem is particularly prominent in offline RL where high-reward trajectories are sparse and affect both policy learning and policy evaluation. To explore this hypothesis, we consider the influence of the train/validation partition in the same sepsis domain, and we evaluate the trained policies using the Weighted Importance Sampling (WIS) (Precup, 2000) estimator. Figure I(a) shows the policies have drastically different OPE estimations with sensitivity to randomness in the dataset partitioning. We can observe the same phenomena in Figure 1(b) with largely different true performances depending on the dataset splitting for most of the policies A,. This also illustrated on the left sub-figure of Figure where in the case where single train-validation split is used, an A, that yields lower- performing policies will often be selected over those that yield nigherr-erfonnen policies when deployed.
4.1 Repeated Experiments for Robust Hyperparameter Evaluation in Offline RL
We now demonstrate why it is important to conduct repeated random sub-sampling on the dataset in offline RL. Consider finite set of J offline RL algorithms A. Let the policy produced by algorithm A; on training dataset D be Tī. its estimated performance (unknown) value be VT Denote the true best resulting policy as πÿ = arg max, V and the corresponding algorithm Aj-. Let the best policy picked based on its validation set performance as T3. arg max, VTa and the corresponding algorithm Az..
validation set V and its true
Theorem 1. There exist stochastic decision processes and datasets such that (i) using a single train/validation split procedure that selects algorithm-hhsperprammn with the best performance

---

on the validation dataset will select suboptimal policy and algorithm with significant finite prob- ability, p + TTz)) 2 C. with corresponding substantial loss in performance O(Vmoz), and. in contrast, (ii) selecting the algorilm-hhyerpareeeete with the best average validation performance across N₂ train/validation splits will select the optimal algorithm and policy with probability 1:
- P(mg. = =T(z) ++1. 1.
Proof Sketch. Due to space constraints we defer the proof to Appendix A.3. Briefly, the proof proceeds by proof by example through constructing a chain-like stochastic decision process and considers class of algorithms that optimize over differing horizons (see e.g. Jiang et al. (2015); Cheng et al. (2021); Mazoure et al. (2021)). The behavior policy is uniformly random meaning that trajectories with high rewards are sparse. This means there is a notable probability that in single partition of the dataset, the resulting train and/or validation set may not contain high reward trajectory, making it impossible to identify that a full horizon algorithm, and resulting policy, is optimal.
In the proof and our experiments, we focus on when the training and validation sets are of equal size. If we use an uneven split, such as 80/20%, the failure probability can further increase if only a single partition of the dataset used. We provide an illustrative example in the Appendix. Note that Leave-one-out Cross- Validation (LooCV) will also fail in our setting if we employ, as we do in our algorithm, WIS, because policy if averaging over a single trajectory, independent of the target policy to be evaluated. We explain this further in Appendix A.11.
biased estimator, WIS will return the observed return of the behavior
5 SSR: Repeated Random Sampling for A, Selection and Deployment
In this paper, we are interested in the following problem: If offline RL training and evaluation are very sensitive the partitioning of the dataset, especially in small data regimes, how we reliably produce a final policy that we are confident is better than others and can be reliably deployed in the real-world?
Instead of considering the sensitivity to data partition as an inherent obstacle for offline policy selection, we view this statistics to leverage for A, selection. We propose a general pipeline: Split Select Retrain (SSR) (of which we provide a pseudo-code in Algorithm Appendix A.4) to reliably optimize for a good deployed policy given only: an offline dataset, an input set of AH pairs and an off-policy evaluation (OPE) estimator. This deployment approach leverages the random variations created by dataset partitioning to select algorithms that perform better on average using a robust hyperparameter evaluation approach which we develop below.
First, we split and create different partitions of the input dataset. For each train/validation split, each
(AH) is trained on the training set and evaluated using the input OPE method to yield an estimated value the validation set. These estimated evaluations are then averaged, and the best AH pair (A4") is selected as the one with the highest average score. Now the last step of the SSR pipeline is to re-use the entire dataset to train one policy x² using A".
Repeated Random Sub-sampling (RRS). As Theorem suggests, one should ensure a sufficient amount of trajectories in the evaluation partition to lower the failure rate C. We propose to create RRS train-validation partitions. This approach has many names in the statistical model selection literature. such as Predictive Sample Reuse Method (Geisser, 1975), Repeated Learning- Test Method (Burman, 1989) Monte-Carlo Cross-Validatie Validation (Dubitzky et al., 2007). It has also been referred to as Repeated Data Splitting (Chernozhukov et al., 2018) 8) in the heterogeneous treatment effect literature.
We randomly select trajectories in D and put them into into two parts: Atrain and Avalid We repeat this splitting process K times to generate paired datasets: (Rirain Ryaidd), (Riraann Ryalid) ) ...., (R. (Rigain Realid). We compute the generalization performance estimate as follows:
GARRSK
1 K
Σ
[VA((Rrain) Ryalid)]
k=
key advantage of overlap partitioning is that it maintains the size of the validation dataset as K increases. This might be favorable since OPE estimates highly dependent on the state-action coverage of the validation dataset the more data in the validation dataset, the better OPE estimators can evaluate policy's 's performance. As K + so, RRS approaches the leave-p-out cross-vale coss-validation
5

---

(CV), where denotes the number of examples in the validation dataset. Since there are (x) possible selections of data points out of in our dataset, it is infeasible to use exact leave-p-out CV when p > 2, but finite K can still offer many advantages. Indeed, Krzanowski and Hand (1997) point out that leave-p-out estimators will have lower variance compared to leave-one-our estimators, which what the more commonly used M-fold cross-validation method converges when M 1. We discuss more in Appendix A.2.
Experiments
In this section, we answer the following questions: (a) how does the pipeline SSR-RES compare to other methods? (b) does the proposed pipeline for A, selection and policy deployment allow to generate the best policy trained on the whole dataset? (c) does re-training on the whole dataset (data efficiency) generate better policies than policies trained on half of the dataset when A; is selected by the pipeline? In addition, we conduct two ablation studies to answer to: what number of splits should we use for SSR-RRS, and what is the impact of dataset size on the pipeline results?
6.1 Task/Domains
The experimental evaluation involves a variety of real-world and simulated domains, ranging from tabular settings to continuous control robotics environments. We evaluate the performance of SSR in selecting the best algorithm regardless task domains and assumptions on task structure. We conduct experiments on eight datasets (Figure 2) from five domains (details in Appendix A. A.14), which we give a short description below, and use as many as 540 candidate AH pairs for the Sepsis POMDP domain.
Intravenous Veroppeisiorr
Fludd
Figure 2:
Illustrations from left to right of the D4RL, Robomimic, TutorBot and Sepsis domains.
Sepsis. The first domain is based on the simulator and work by Oberst and Sontag (2019) and revolves around treating sepsis patients. The goal of the policy for this simulator is discharge patients from the hospital. In this domain, we experiment on two tasks: Sepsis-MDP and Sepsis-POMDP, a POMDP version of Sepsis-MDP.
The second domain includes a TutorBot simulator that is designed to support 3-5th grade elementary school children in understanding the concept of volume and engaging them while doing RL agent, which interacted with
TutorBot.
so. An online study was conducted using policy-gradient-beee about 200 students. We took the observations from this online study and built simulator that reflects student learning progression, combined with some domain knowledge.
Robomimic. Robomimic (Mandlekar et al., 2021) is composed of various continuous control robotics environments with suboptimal human data. We use the Can-Paired and Transport dataset composed of 200 mixed-quality human demonstrations. Mandlekar et al. (2021) attempted use the RL objective loss on 20% split validation set to select the best AH pair, but reported that the selected AH did not perform well in the simulator, which makes this task an interesting testbed for our pipeline.
D4RL. D4RL (Fu et al., 2020) is offline RL standardized benchmark designed and commonly used evaluate the progress of offline RI. algorithms. We use 3 datasets (200k samples each) with different qualities from the Hopper task: hopper-random from a randomly initialized policy, hopper- mepper- from a policy trained to approximately 1/3 the performance of a policy trained to completion with SAC ("expert"), and mawtrer-ediu-eeppeeeee from 50-50 split of medium and expert data.
6.2 Baselines
One-Split OPE. The simplest method to train and verify an algorithm's performance without access to any simulator is to split the data into train Dtrain and valid set Dvalid- All policies trained

---

PPMDI N=1000
Sopsis-POMDP
Scpsis-MDP N--00
Robomimic Can-Paired N=200
Robomimic Transport N-200
Palicy Performance in Real World
8-
Oars
7.86
N-1000
6.57
Nested CV
0.5
65-
OPE Bonidrapped Val.
4,30
-10
Validation
Choss
-35
04
2.20
-8-
0:3
BVFL
0.2-
-12
0.1
122 122
-18
55%
100%
30%
100%
50%
100%
One-Spiit OPE
BCa (mean)
Bca (UB)
BCa (LB)
BVFT-PGE
Cv 2-fold
CV f-fold
Nested 2x2
Nested 5x2
SSA RAS-2
SSR RAS-S
Optimal
(b)
Figure 3: (a) We first compare our proposed pipeline to various other policy selection approaches in the Sepsis POMDP task. Our approach SSR-RRS 5-split consistently obtains policies that on average perform close to the optimal policy, significantly outperforming other approaches. the importance of re-training in the small (N=200) to medium (N=1000) data regime. We show the true reward obtained by all policies from all AH pairs either trained on 50% of the data or 100% of the data. 95% confidence intervals depicted as error bars. Most policies achieve higher rewards when trained on more data, even more so when the dataset is small or when tasks are more difficult (Robomimic).
(b) We investigate
on the same training set and evaluated on the same valid set. As we explained before, this method has the high potential of overfitting the chosen hyperparameter for one data partition - it might pick the best policy, but does not guarantee we can use the same hyperparameter to re- re-train on the full dataset.
OPE on Bootstrapped Val. Bootstrapping is a popular re- sampling technique that estimates pre- diction error in supervised learning models (Efron, 1983, 1986). The idea of using bootstrapping for OPE estimate is first utilized in HCOPE (Thomas et al., 2015b). Compared to the one-split method and the SSR pipeline. bootstrapping trains all policies on the same training dataset, and only considers variations in the validation set by creating bootstrapped samples. We refer to the considered Bias-corrected accelerated (BCa) bootstrap method as BCa in the experiments.
Vols--aaldation (CV). One other natural alternative of repeated experiment validation is the popular M-fold Cross Valiiaaiin method (Stone, 1974). M-fold CV constructs a non-overlapping set of trajectories from the original dataset. For example, a 5-fold CV will train a policy on 80% of data and evaluate the policy on 20% of data, as it divides the dataset into non-overlapping partitions. However, as we increase the number of splits M, which allows us to train/test our algorithms under more data split variations, each non-overlapping set D,,. becomes smaller. When M CV becomes leave-one-out CV (LooCV). In this extreme case, many OPE estimators will not work properly, as we have shown in Appendix A.3. We further investigate variant of M-fold CV called Nested M-fold CV (Nested CV), which repeats the M-fold non-overlapping partitioning K times. This procedure is computationally very expensive. Considering the fairnsss of comparison and computational efficiency, we only evaluate K x 2-fold CV.
M-fold
OPE with BVFT. Batch Value Function Tournament is the closest competitor to our method, which is a meta-algorithm for hyperparameter- free policy selection (Xie and Jiang, 2021; Zhang and Jiang, 2021). For set of Q-value functions, BVFT makes pairwise comparisons of each (ourmaaenn--stylee to select the best out of the entire set based on the B BVFT-Loss. Compared to our method, BVFT incurs (J)²) comparison given J AH pairs, practically infeasible for large J. The original BVFT can only compare Q-functions, therefore only usable with OPL. that directly learns Q-functions. Zhang and liang (2021) offers extension to BVFT by using BVFT compare between FQEs, therefore allowing BVFT to be OPL-agnostic. We adopt the two strategics recommended by the paper. Given J AH pairs and B FQEs, strategy I compares J x B FQE's Q-functions jointly (7 x FQE) and strategy 2 compares B FQEs within each AH and pick the best FQE as the estimate of the AH's value estimate (I + FQE). We discuss more in Appendix A.6 (calculations) and A.7 (time complexity).
6.3 Training and Evaluation
Offline Policy Learning. In the following, we outline a variety of Offline RL algorithms used in the evaluation of the SSR pipeline to demonstrate the generality of our approach and that it can reliably
7

---

Internal Q-function
Category
Algorithms
BC (Pomerleau, 1991)
Imitation Learning
x
BCRNN (Mandlekar et al., 2018)
BCQ (Fujimoto et al., 2019). CQL (Kumar al., 2020).
Conservative Model-Free
/
IRIS (Mandlekar et al., 2020)
Policy Gradient
POIS (Metelli et al., 2018), BC+POIS (ours), BC+mini- (ours)
x
Offline Model-Based
MOPO (Yu et al., 2020), P-MDP (ours)
/
Table List of AH we are comparing our experiments.
One-Split OPE
SSR RRS-2
SSR RRS-5
bonk
10-
Dii
bog mp. 17-
73
-20
Dif In Picked Policy
SSA (52% Datal
SSA RAS (SAR) Deta)
reto mp
OPf (50%) Data)
OPE with (L90% Dotb)
(19h Perteraace
1191--20 Noffrmancc
W0 Sode Perfommrce
N--00
N-1000
N-5000
Low performing policies
High performing policies
(a)
(b)
Figure 4: (a) We show the effect of choosing K (the number of train/valid splits) for SSR-RRS. We ablate K and run the simulation 500 times. The heatmaps shows the frequency which each policy is chosen by our method and its final performance in the environment. In this experiment, SSR-RRS chooses over 540 AH pairs. As we can see, when the number of splits K' is larger, SSR-RRS consistently pick better policies. (b) In the Sepsis-POMDP domain, we show that as the number of trajectories (N) in the offline dataset increases, data partitioning becomes less important. Though RRS still outperforms the 1-split policy selection.
produce the optimal final policy using the selected AH pair. This marks a departure from workflows designed for specific algorithms, such Kumar et al. (2021). We experiment with popular offline RL methods (see Table 2 and we provide algorithmic and hyperparameter details in Table A.2). Offline Policy Evaluation. We use WIS estimators for the tabular and discrete action domains: Sepsis and TutorBot. We use FQE for continuous action domains: Robomimic and D4RL. For each task, with a given dataset, we use the splitting procedure described in Section 5 to generate the partitioning. We describe how we compute the results for each figure Appendix A.13.
7 Results
A; Selection Comparison. In Figure 3(a), we compare five approaches (One-Split OPE, BCa, BVFT-FQE, CV, and SSR-RRS) on the Sepsis-POMDP domain with 1000 patients. For each approach, we compute score per AH pair, and select the best algorithm according to each. For fairness in comparison, all selected A, are re-trained on the full dataset and we report the final performance in the real environment. As expected, One-Split OPE performed the worst. Surprisingly, using the lower bound of bootstrapped (BCa (LB)) confidence interval also does not allow pick good policies, LB being perhaps too conservative. We see that CV 2-fold and CV 5-fold do not perform well either. CV 2-fold does not allow enough repetition and CV 5-fold makes the validation set size too small. We observe clearly that SSR-RRS 5-split performs the best and selected policies that are on average very close to the optimal policy's performance. BVFT-FQU relies on FQE, which is misspecified model on the Sepsis domain and difficult to optimize given the small dataset size; hence it does not select good policies in Sepsis. However, in Robommmic Can-Paired and D4RL Hopper, BVFT-FQE is able pick good policies, albeit not significantly better or worse than other methods, and still worse than SSR-RRS 5-split in the mixed (more realistic) "medium-expert" dataset. We show more analysis of BVFT compared to method in the Appendix. Table 3 aggregates the results for all the considered domains in our study. Our approach SSR-RRS 5-split is distinctly able to more consistently select policies that, once deployed, perform close to the optimal policy across all tasks.
8

---

Re-trained full dataset
BVFT-FQE क FQE
BVFT-FQE FQE
SSR RRS-2
SSR RRS-5
Optimal Policy
CV-2
CV-5
Robomimic:
Can-Paired
0.65
0.71
0.72
0.72
0.71
0.73
0.75
Transport
0.21
0.42
0.42
0.62
0.70
0.74
D4RL (Hopper):
random
321.75
317.72
325.37
325.37
324.92
325.37
325.37
medium
934.71
1227.81
1227.81
1304.53
1296.87
1304.54
1392.93
medium-expert
2677.93
2677.93
2530.04
2530.04
3481,34
3657.80
3657.80
Sepsis: MDP (n=200) POMDP (n=1000)
-19.32
-10.26
-20.32
-13.01
-7.85
-1.94
-1.92
0.74
-1.92
2.40
6.75
7.86
TutorBot:
POMDP (n=200)
1.34
1.19
1.30
1.38
1.43
Table 3: Comparison of the performance obtained by policy deployed using the SSR pipeline vs. using 1-split policy selection approaches on wide range of application domains. Cells average true return. We note that (æ FQE) is very computationally expensive when we search through large space (in Sepsis and TutorBot), therefore we exclude them.
The Benefits of Re-training Policies Selected with SSR, In Figure 3(b). we plot the true reward of selected policy A, when only trained on 50% of the dataset (the training set) compared to when trained on 100% of the dataset. As expected, in the small data regime, every single trajectory matters. Policies trained on the full dataset significantly outperform policies trained only on half of it. This experiment provides strong evidence favor of AH selection (done with RRS on the full dataset) over policy selection (done on the training set) in offline RL.
The Impact of Number of Repeats for SSR-RRS. The proposed pipeline SSR-RRS has hyperparam- eter for the number of repeated data splitting. In Figure 4(a), we show the true performance the policy that is being selected by SSR-RRS with K = 1, 2, 5 by running 500 simulations with heatmaps on the frequency of each policy is selected. We observe that when K (equivalent the One-Split OPE method), policies are picked quite uniformly; many of which are performing poorly. When K 5, higher-perfoee perforrinn policies selected much more frequently. From Table 3, we conclude that = generally works well across various domains. Naturally, the number of split K will be chosen in line with the computing budget available; K 5 appears to be reasonable choice.
The Impact of Dataset Size. Finally, investigate which extent the proposed pipeline is nec- essary when the dataset size is sufficiently large. We use the Sepsis-POMin POMDP domain with 200, 1000, and 5000 patients. We show the best policies that are most frequently selected by approach in Fig- ure 4(b). Unsurprisingly, policies trained on larger datasets perform better. In the 200-patient dataset, having SSR-RRS 5-split is crucial in picking the best policy, most policies perform quite poorly. The gap between different approaches becomes smaller with 1000 patients, and even smaller when there are 5000 patients in the dataset. However, it is worth noting that even in the large dataset regime (N=5000), SSR-RRS still outperforms the One- One-Split OPE method in selecting the best algorithm.
Additional Analysis. Our method SSR-RRS can also be used to select hyperparameters for a single algorithm, as we demonstrate in Appendix A.9, One might also wonder how sensitive is SSR- RRS pipeline to the choice of OPE method used inside the pipeline. OPE methods are known to significantly vary in accuracy for different domains, and unsurprisingly. using reasonable OPE method for the domain important (see Appendix A.8). Note that the OPE estimators we use in our results are very popular ones, and is possible to use standard approaches, though additional benefits may come from using particular OPE methods might be biased towards certain OPL algorithms which make similar assumptions (such as assuming Markov structure): interestingly in preliminary experiments, FQE estimators did not seem to give FQI algorithms higher performance estimations (see Appendix A.10).
better OPE methods. Finally, related to this question, might wonder
8 Discussion and Conclusion
We presented SSR, a pipeline for training, comparing, selecting and deploying offline RL policies in small data regime. The approach performs automated AH selection with robust hyperparameter

---

evaluation process using repeated random sub-sampling, SSR allows to consistently and reliably deploy best-performing policies thanks to jointly avoiding overfitting on a single dataset split and being data efficient in re-using the whole dataset for final training. We prove that single split has a high failure rate of discovering the optimal AH because of reward sparsity. We have demonstrated its strong empirical performance across multiple and various challenging domains, including real- world applications where AH tuning cannot be performed online.
There exist many interesting areas for future work. The proposed offline RL pipeline assumes the user/practitioner has selected a particular OPE method. OPE is an important subarea of its own and different approaches have different bias/variance tradeoffs. Recent work automated model selection algorithms for OPE (Su et al., 2020; Lee et al., 2021) are a promising approach for producing good internal estimators. A second issue is that while our approach aims to produce a nigh-performing policy, it does not also produce an accurate estimate of this policy since the entire dataset used at the end for training. An interesting issue is whether cross- solssssplitiing (Chernozhukov et al., 2016) or other methods could be used compute reliable estimators as well as perform policy optimization.
Acknowledgment
Research reported in this paper was supported in part by a Hoffman- Yee grant, NSF grant #21 12926 and the DEVCOM Army Research Laboratory under Cooperative Agreement W91 W911NF-1 17-2-0196 (ARL IoBT CRA). The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed implied, of the Army Research Laboratory or the ..S.Govvrrmeet.. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein. We would like to thank Jonathan N. Lee, Henry Zhu, Matthew Jorke, Tong Mu, Scott Fleming, and Eric Zelikman for discussions.
References
Bassen, J., Balaji, B., Schaarschmidt, M., Thille, C., Painter, J., Zimmaro, D., Games, A., Fast E., and Mitchell, J. C. (2020). Reinforcement learning for the adaptive scheduling of educational activities. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pages 1-12.
Burman, P. (1989). A comparative study of ordinary cross-validation, v-fold cross-validation and the repeated learning-testipg methods. Biometrika, 76(3):503-511
Cheng, C.-A., Kolobov, A., and Swaminathan, A. (2021). Heuristic-guided reinforcement learning. Advances in Neural Information Processing Systems, 34.
Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., and Robins, J. (2016). Double/debiased machine learning for treatment and causal parameters. arXiv preprint 1608
Chernozhukov, V.. Demirer, M., Duflo, E., and Fermandez-Vall I. (2018). Generic machine learning inference on heterogeneous treatment effects in randomized experiments, with an application to immunization in india. Technical report, National Bureau of Economic Research.
Dietterich, T. G. (1998). Approximate statistical tests for comparing supervised classification learning algorithms. Neural computation, 10(7):1895-19)
Dubitzky, W., Granzow, M., and Berrar, P. (2007). Fundamentals of data mining in genomics and proteomics. Springer Science & Business Media.
Efron, B. (1983), Estimating the error rate of a prediction rule: improvement on cross-validation. Journal of the American statistical association, 78(382):31633311
Efron, B. (1986). How biased is the apparent error rate of a prediction rule? Journal of the American statistical Association, 816394)4461-470.
Farajtabar, M., Chow, Y., and Ghavamzadeh, M. (2018). More robust doubly robust off-policy evaluation. In International Conference Machine Learning, pages 1447-1456. PMLR.
10

---

Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. (2020). D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219.
Fujimoto, S., Meger, D., and Precup, D. (2019). Oft- Off-policy deep reinforcement learning without
exploration. In International Conference on Machine Learning, pages 2052-2062. PMLR.
Futoma, J., Hughes, M. C., and Doshi- Velez, F. (2020). Popcorn: Partially observed prediction constrained reinforcement learning. arXiv preprint arXiv:2001.040:2.
Geisser, S. (1975). The predictive sample reuse method with applications. Journal of the American statistical Association, 706350):320-328.
Gilpin, A. R. (1993). Table for conversion of kendall's tau spearman's 's tho within the context of measures of magnitude of effect for meta-analysis. Educational and psychological measurement, 53(1):87-92.
Guo, Y., Feng. S., Le Roux, N., Chi, E., Lee, H., and Chen, M. (2020). Batch reinforcement learning through continuation method. In International Conference on Learning Representations.
Jiang, N., Kulesza, A., Singh, S., and Lewis, R. (2015). The dependence effective planning horizon model accuracy. In Proceedings of the 2015 International Conference Autonomous Agents and Multiagent Systems, pages 1181-1189, Citeseer.
Kidambi, R., Rajeswaran, A., Netrapalli, P., and Joachims, T. (2020). Morel: Model-based offline reinforcement learning. Advances in neural information processing systems, 000-2180022882
Komorowski, M., Celi, L. A., Badawi, O., Gordon, A. C.., and Faisal, A. A. (2018). The artificial intelligence clinician learns optimal treatment strategies for sepsis in intensive care. Nature medicine,
Krzanowski, W. and Hand, D. (1997). Assessing error rate estimators: the leave-one-out method reconsidered. Australian Journal of Statistics, (91))355-4
Kumar, A., Singh, A., Tian, S., Finn, C., and Levine, S. (2021). A workflow for offline model-free robotic reinforcement learning. In 5th Annual Conference Robot Learning.
Kumar, A., Zhou, A., Tucker, G., and Levine, S. (2020). Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1179-1191.
Le, H., Voloshin, C., and Yue, Y. (2019). Batch policy learning under constraints. In International Conference Machine Learning, pages
PMLR.
Lee, J. N., Tucker, G., Nachum, O., and Dai, B. (2021). Model selection in batch policy optimization. arXiv preprint 12.
Levine, S., Kumar, A., Tucker, G., and Fu, (2020). Offline reinforcement learning: Tutorial, review, and perspectives open problems. arXiv preprint arXiv:2005.01643:
Liao, P., Greenewald, K., Klasnja, P., and Murphy, S. (2020). Personalized heartsteps: A reinforce- ment learning algorithm for optimizing physical activity. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 4(1):1-22.
Liu, Y., Swaminathan, A., Agarwal, A., and Brunskill, E. (2019). Off-policy policy gradient with stationary distribution correction. In Globerson, A. and Silva, R., editors, Proceedings of the Thirty-Fifth Conference Uncertainty in Artificial Intelligence, UAI 2019, Tel Aviv, Israel, July 22-25, 2019, volume 115 of Proceedings of Machine Learning Research, pages 1180-1190. AUAI Press.
Liu, Y., Swaminathan, A., Agarwal, A., and Brunskill, E. (2020). Provably good batch reinforcement learning without great exploration. Advances in neural information processing systems, 33.
Mandlekar, A., Ramos, F., Boots, B., Savarese, S., Fei-Fei, I.., Garg, A., and Fox, D. (2020). Iris: Implicit reinforcement without interaction at scale for learning control from offline robot manipulation data. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 1414-4420. IEEE.

---

Mandlekar, A., Xu, D., Wong, J., Nasiriany, S., Wang, C., Kulkarni, R., Fei-Fei, L., Savarese, S., Zhu, Y., and Martin-Martin R. (2021). What matters in learning from offline human demonstrations for robot manipulation. In 5th Annual Conference on Robot Learning.
Mandlekar, A., Zhu, Y., Garg, A.. Booher, J., Spero, M., Tung, A., Gao, J., Emmons, J., Gupta, A., Orbay, E., et al. (2018). Roboturk: A crowdsourcing platform for robotic skill learning through imitation. In Conference on Robol Learning, pages 879-893. PMLR.
Mazoure, B., Mineiro, P., Srinath, P., Sedeh, R. S., Precup, D., and Swaminathan, A. (2021). Improving long-term metrics in recommendation systems using short-horizon reinforcement learning. arXiv preprint arXi::2106.0055
Metelli, A. M., Papini, M., Faccio, F., and Restelli, M. (2018). Policy optimization via importance sampling, Advances in Neural Information Processing Systems, 31.
Miyaguchi, K. (2022). A theoretical framework of almost hyperparameter--fre hyperparameter selection methods for offline policy evaluation. arXiv preprint arXiv:2201.02300.
Oberst, M. and Sontag, D. (2019), Counterfactual off-policy evaluation with gumbel- structural causal models. In International Conference on Machine Learning, pages 4881-4890. PMLR.
Owen, A. B. (2013). Monte carlo theory, methods and examples.
Paine, T. L., Paduraru, C., Michi, A., Gulcehre, C.,, Zolna, K., Novikov, A., Wang, Z., and de Fre- itas, N. (2020). Hyperparameter selection for offline reinforcement learning. arXiv preprint arXiv:2007.09055
Pomerleau, D. A. (1991). Efficient training of artificial neural networks for autonomous navigation. Neural computation, 3(1):88-97.
Precup. D. (2000). Eligibility traces for off-policy policy evaluation. Computer Science Department Faculty Publication Series, page 80.
Shi, C., Wan, R., Song, R., Lu, W., and Leng. L. (2020). Does the markov decision process fit the data: Testing for the markov property in sequential decision making. In International Conference on Machine Learning, pages 8807-8817. PMLR.
Siegel, N., Springenberg. J. T., Berkenkamp, F., Abdolmaleki, A., Neumert, M., Lampe, T., Hafner, R., Heess, N., and Riedmiller, M. (2020). Keep doing what worked: Behavior modelling priors for offline reinforcement learning. International Conference on Learning Representations.
Stone, M. (1974). Cross-validatory choice and assessment of statistical predictions. Journal of the royal statistical society: Series B (Methodological), 36(2):111-133.
Su. Y., Srinath, P., and Krishnamurthy, A. (2020). Adaptive estimator selection for off-policy evaluation. In International Conference on Machine Learning, pages 9196-9205. PMLR.
Tang. S. and Wiens, J. (2021). Model selection for offline reinforcement learning: Practical con- siderations for healthcare settings. In Machine Learning for Healthcare Conference, pages 2-35. PMLR.
Thomas, P. and Brunskill, (2016). Data-efficient off-policy policy evaluation for reinforcement learning. In International Conference on Machine Learning, pages 2139-21483 PMLR.
Thomas, P., Theocharous, G., and Ghavamzadeh, M. (2015a). High confidence policy improvement. In Bach, F. and Blei, D., editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 2380-2388 Lille, France. PMLR.
Thomas, P., Theocharous, G., and Ghavamzadeh, M. (2015b). High confidence policy improvement. In International Conference on Machine Learning, pages 2380-2388. PMLR.
Thomas, P. S., Castro da Silva, B., Barto, A. G., Giguere, S., Brun, Y., and Brunskill, E. (2019). Preventing undesirable behavior of intelligent machines. Science, 366(6468))999-100
12

---

Voloshin, C., Le, H. M., Jiang, N., and Yue, Y. (2021). Empirical study of off- policy evaluation for reinforcement learning. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1).
Xie, T. and Jiang, N. (2021). Batch value-function approximation with only realizability, In International Conference Machine Learning, pages 11404 11413, PMLR.
Yin, M., Bai, Y., and Wang, Y.-.. (2021). Near-optimal offline reinforcement learning via double variance reduction. Advances in neural information processing systems, 34.
Yu, T., Thomas, G., Yu, L.. Ermon, S., Zou, J. Y., Levine, S., Finn, C., and Ma, T. (2020). Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33:14129-1444 14142.
Zanette, A. and Brunskill, E. (2019). Tighter problem-dependent regret bounds reinforcement learning without domain knowledge using value function bounds. In International Conference Machine Learning, pages 7304-731 PMLR.
Zhang, and Jiang, N. (2021). Towards hyperparameter-free policy selection for offline reinforce- ment learning. Advances in Neural Information Processing Systems, 34.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you deseribe the limitations of your work? [Yes] See Section 7 and Section 8.
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section 4, and Section A.3.
(b) Did you include complete proofs of all theoretical results? [Yes] See Section 4. and Section A.3.
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material as URL)? [Yes] See Supple- mentary Material.
(b) Did you specify all the training details (c.g., data splits, hyperparameters, how they were chosen)? [Yes] See Sections 6, A. 14, A.16, A.17, A.18 and A.19
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes)
(b) Did you mention the license of the assets? [N/A] Open-source
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "znYUQvlB1VQ", "reviewer": "Reviewer_CB5Q", "summary": "The paper proposes SSR, a new pipeline for selecting the best-performing deployment policy in the offline RL setting. The key idea of SSR is the use of repeated random sampling (RRS), which samples $K$ overlapping train-test pairs from the offline dataset. Each specific algorithm-hyperparameter choice is trained and evaluated $K$ times using these pairs. SSR picks the policy with the highest average performance. The authors justify SSR both theoretically and empirically. SSR consistently outperforms other policy selection methods over various environments, especially in low-data regime.", "questions": "- Take computation aside, will increasing $K$ always improve the performance of the final policy?\n\nMinor questions/suggestions:\n- Legend of Figure 4(b) is confusing, is there a typo there?", "limitations": "The authors have adequately addressed the limitations of the work.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- The paper tackles the problem of hyperparameter tuning, or policy selection, which is a relevant and essential aspect of offline RL.\n- To the best of my knowledge, the idea presented is original. While repeated random sampling is not a new technique, it has never been tried in the offline RL setting.\n- The idea is simple and well motivated. The authors did a good job of pointing out the main limitation of current OPE and OPL methods: the high variance with respect to the train-validation split. SSR naturally solves this issue by performing this split multiple times and averaging the performance.\n- The authors evaluate SSR on extensive benchmarks, ranging from robotics to medical domains. SSR consistently produces better policies than other selection methods.\n- The paper is easy to read in general.\n\n### Weaknesses\n- My first concern regards the robustness of SSR with respect to the chosen offline RL algorithm. In the paper, the authors only show the performance of the best policy among all AH pairs. However, it is also interesting to see if SSR consistently performs well given a specific algorithm. This reduces policy selection to hyperparameter tuning, which is important for future research in offline RL.\n- The second concern regards the effectiveness of SSR with respect to the choice of OPE. Will there be an OPE that is robust enough to the partitioning of the dataset that using repeated sampling will not help? I would love to see at least one more OPE being used in SSR in addition to WIS or FQE.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "FwLLVh6ke3I", "reviewer": "Reviewer_x8P4", "summary": "This paper intends to improve offline reinforcement learning (RL) with limited data from a data-partition perspective. The authors first analyze why traditional data splittings don't perform well, including non-splitting (on the entire dataset) and cross-validation splitting. Finally, the authors propose a repeating random sampling strategy to enhance the selection of algorithm-hyperparameter (AH). The authors design experiments to demonstrate the superiority of the proposed SSR-RRS method in terms of policy performance.", "questions": "1. In line 37, the authors claim that the trained policies cannot be well evaluated, when the validation set has limited high-reward trajectories. This is doubtful, because the training set is responsible for the learning process, while the validation set is only used to 'predict'. It seems that after the policy is trained on the training set, its 'prediction' capability is not affected by the validation set. Is there any justification for that?\n2. In RRS, can the policies trained by previous splittings be used as the pre-training for current splitting?", "limitations": "There is no apparent limitation on the work. It is universal and can be adapted to different offline RL algorithms. It will have a broad impact on the RL area.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The research topic, improving offline RL from the data splitting angle, is inspiring and innovative. Data splitting is widely investigated in supervised learning, but it is rarely studied in the RL area. This paper proves that data splitting has a significant impact on offline RL training.\n2. The idea of repeated random sub-sampling (RRS) is novel. The RRS strategy integrates ideas of ensemble learning and cross-validation, which is maybe not very fancy, yet very innovative and effective.\n3. Almost all claims and theorems are justified either from the theoretical or the empirical analysis, making the paper solid and sound.\n\n\nWeaknesses:\n1. The parameter sensitive analysis for the RRS method is not comprehensive. To make the work more practical, the readers would like to know how to choose a reasonable splitting number $K$. The experiments show that when $K=5$ has the best performance. Does this mean the larger $K$ the better? Will it drop when reaching some thresholds?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "FfjDBvuIJa", "reviewer": "Reviewer_DpHr", "summary": "The paper proposes a method for offline policy selection for offline reinforcement learning algorithms. The main idea is to use repeated random splitting of the dataset to pick the best hyper-parameters before training on the entire dataset. The algorithm is evaluated against multiple datasets and compared against standard benchmarks. ", "questions": "Q1. The proof of theorem 1 assumes that the behavior policy is uniform random, and that high reward trajectories are sparse. This may not be true in practice. Historical data is usually from a non-random policy already deployed in a system. Can you explain why this assumption holds, and what are the implications if it is voided.\n\nQ2. In line 174, you claim \"WIS will return the observed return of the behavior policy if averaging over a single trajectory, independent of the target policy to be evaluated.\" It is not clear why this is true. There is no evidence, analysis or citation. \n\nQ3. Line 207 claims: \"As K → ∞, RRS approaches the leave-p-out cross-validation (CV), where p denotes the number of examples in the validation dataset.\" Again, not clear why this is true. How does p relate to the size of the dataset you use in RRS (split by 2)? ", "limitations": "The entire paper relies on a single evaluation method for a dataset (WIS for discrete, FQE for continuous MDPs). It would be good to see if the claims hold for another evaluation method. ", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper tackles an important problem in offline reinforcement learning. Without a policy selection method, it is likely that the resulting policies perform poorly after deployment. \n- The algorithm proposed is well motivated with theory and compared with the related works in literature.\n- The evaluation procedure is systematic, with datasets from multiple domains in reinforcement learning and comparisons against the state-of-the-art policy selection algorithms. \n\nWeaknesses:\n- Some of the claims in the paper do not have evidence or reasoning. Details below. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "FBU-LtmQLn2", "reviewer": "Reviewer_g82q", "summary": "This paper proposes a workflow for hyperparameter and algorithm selection in offline RL. The authors argue that because datasets can be small and high variance, and algorithms themselves are hyperparameter sensitive, the prevailing approach of training on a fixed training set and tuning on a validation set is not sufficient. The authors instead argue to use re-sampling of the dataset and then average OPE estimates on the validation set in order to estimate the performance of the algorithm & hyperparameters. The authors evaluate on a collection of tasks (based on D4RL, MIMIC-III Sepsis, Robomimic, and Tutorbot) where they show their selection method typically finds the best policies compared to the baselines.", "questions": "- What was the computational cost of the various approaches used in evaluation? While SSR-RRS 5 produces the best results, does it also require significantly more compute that other approaches? \n- My intuition is that OPE is fairly inaccurate in predicting the values of the policies, esp on high-dimensional domains. Was the accuracy of OPE a bottleneck for the method, or did it general predict the relative ordering of policies correctly?", "limitations": "The authors did not discuss potential negative societal impacts of their work, but they discussed a reasonable set of limitations for their method.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- The evaluation is well-done and convincingly shows the utility of the proposed approach. The authors used several logical baselines and several domains that are fairly different qualitatively (ranging from healthcare to robotic control). \n- This work has potential to be fairly significant within the field of offline RL, since it is applicable to most algorithms, and performs better than the common naive approach to hyperparameter selection.\n- The writing is clear and well-organized, The authors motivate the use of their method from theory, describe their practical approach, and clearly explain the rationale behind their chosen baselines.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "EwYfNI7wof-", "reviewer": "Reviewer_5T2Y", "summary": "This paper studies the model and hyper-parameter selection in offline RL, which is an important problem as most offline RL algorithms are sensitive to the algorithm it used, the hyper-parameters and other subtle algorithm details. Different from supervised learning, OPE estimate on the validation set is not reliable when the sample size is small and the trajectory is long. This paper proposes SSR (Repeated Random Sampling) for algorithm selection, the idea is simple in which we just randomly split the trajectories as train/val multiple times, and perform algorithm training and OPE on the train/val separately. The algorithm configuration that achieves the highest average performance on the val will be selected, and finally re-train the whole dataset using this selected configuration. Empirical studies on different domains, such as healthcare, robotics show better performance compared with other baselines, such as BVFT, CV. ", "questions": "[1]. I appreciate the effort on making the pipeline algorithm agnostic. One thing for further understanding this, does the author observe any correlation about the training algorithm used and the OPE used for selection? Say is FQE performs better on FQI algorithms in terms of selection compared with WIS? Is there any interplay on this?\n\n[2]. The empirical study tries to span different domains, which is very nice. It would be very helpful if the authors try to organize them in a way that shows the properties of these MDP/POMDP problems, such as sto/det reward, transitions, sparse/dense reward, horizon length etc and the dataset composition: narrow or diverse. It might be interesting to see how the method works across these different settings, and provides some practical recommendations\n\n[3]. Any commends/ideas on reducing the time complexity of the algorithm, as it involves training for multiple times.\n\n\nMinor:\n\n[1]. What are the y-axis in Fig 4(a)? It seems we have 540 AH pairs, but the y-axis is much smaller than this.\n[2]. Is there a typo on the legend of Fig 4(b)? SSR-RRS 5 is missing.", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n[1]. The paper takes a step towards an important problem in offline RL, i.e., the hyper-parameter selection. As most offline RL algorithms are very sensitive to this and it is important to understand the rule/pipeline for how to effectively select the best configuration for the training algorithm.\n\n[2]. The method proposes is simple and well-motivated, especially on how it compared with K-fold CV. \n\n[3]. The ablation study is well-done, especially on the importance of re-train procedure, the impact of the repeats training and the effect of the datasize.\n\nWeakness:\n\n[1]. This method somehow sounds like more of a heuristic. Though Theorem 1 tries to show there exists a setting that this k-split average performance is better than the 1 split, it does not show the correctness of the method, and this result seems weak for the theoretical justification. \n\n[2]. If I understand it correctly, the computational burden for the method seems high, as it needs to retrain for K times, and the training for RL is very time-consuming.\n\n[3]. The paper frames it as an overall pipeline, it discusses more on train/val split, however for the OPE estimate part is missing. How is the pipeline sensitive to the OPE estimate used? Is there any ablation study on this? In the experiments, it just uses the default setting, WIS for discrete and FQE for continuous task.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept