Pessimistic Model Selection for Offline Deep Reinforcement Learning
Authors: Chao-Han Huck Yang
Zhengling Qi
Yifan Cui
Pin-Yu Chen
Venue: ICLR 2022 Submitted (2022)
Keywords: reinforcement learning theory
offline deep reinforcement learning
model selection
pessimism
tuning free

Abstract: Deep Reinforcement Learning (DRL) has demonstrated great potentials in solving sequential decision making problems in many applications. Despite its promising performance,  practical gaps exist when deploying DRL in real-world scenarios. One main barrier is the over-fitting issue that leads to poor generalizability of the policy learned by DRL. In particular, for offline DRL with observational data, model selection is a challenging task as there is no ground truth available for performance demonstration, in contrast with the online setting with simulated environments. In this work, we propose a pessimistic model selection (PMS) approach for offline DRL with a theoretical guarantee, which features a tuning-free framework for finding the best policy among a set of candidate models. Two refined approaches are also proposed to address the potential bias of DRL model in identifying the optimal policy. Numerical studies demonstrated the superior performance of our approach over existing methods.

Under review as a conference paper at ICLR 2022
PESSIMISTIC MODEL SELECTION FOR OFFLINE DEEP REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Deep Reinforcement Learning (DRL) has demonstrated great potentials in solving sequential decision making problems in many applications. Despite its promising performance, practical gaps exist when deploying DRL. in real-world scenarios. One main barrier is the over-fitting issue that leads to poor generalizability of the policy learned by DRL. In particular, for offline DRL with observational data, model selection is a challenging task as there is no ground truth available for per- formance demonstration, in contrast with the online setting with simulated envi- ronments. In this work, we propose pessimistic model selection (PMS) approach for offline DRL with theoretical guarantee, which features a provably effective framework for finding the best policy among a set of candidate models. Two re- fined approaches are also proposed address the potential bias of DRL model in identifying the optimal policy. Numerical studies demonstrated the superior performance of our approach over existing methods.
INTRODUCTION
The success of deep reinforcement learning (Mnih et al. 2013 Henderson al., 2018) (DRL) often leverages upon executive training data with considerable efforts to select effective neural architec- tures. Deploying online simulation learn useful representations for DRL is not always realistic and possible especially in some high-stake environments, such as automatic navigation (Kahn et al.,
2018
Hase et al.) (2020), dialogue learning (Jques et al.. 2020). and clinical applications (Tang
et al.
2020a). Offline reinforcement learning (Singh & Sutton| 1996 Levine et al., 2020 Agarwal 2020) (OffRI.) has prompted strong interests (Paine et al. 2020 Kidambi et 2020) to
et al.
empower DRL. toward training tasks associated with severe potential cost and risks. The idea of OffLL is to train DRL models with only logged data and recorded trajectories. However, with given
observational data, designing successful neural architecture in OnRL is often expensive (Levine
ct al. 2020), requiring intensive experiments, time, and computing resources.
Unlike most aforementioned applications with online interaction, Offline tasks for reinforcement learning often suffer the challenges from insufficient observational data from offline collection to construct universal approximated model for fully capturing the temporal dynamics. Therefore, relatively few attempts in the literature have been presented for provide a pipeline to automate the development process for model selection and neural architecture search in OfRL settings. Here, model selection refers to selecting the best model (e.g., the policy learned by a trained neural net- work) among a of candidate models (e.g. different neural network hyperparameters).
In this work, we propose a novel model selection approach to automate OffRL development process, which provides an evaluation mechanism identify a well-performed DRL model given offline data. Our method utilizes statistical inference to provide uncertainty quantification on value functions trained by different DRL. models, based on which a pessimistic idea is incorporated to select the best model/policy. In addition, two refined approaches are further proposed to address the possible biases of DRL models identifying the optimal policy. In this work, we mainly focus on deep Q-network (Mnih et al. 2013 2015) (DQN) based architectures, while our proposed methods can be extended to other settings. Figure [ demonstrates the superior performance of the proposed pessimistic model selection (PMS) method in identifying the best model among 70 DRL models of different algorithms on one navigation task (See Appendix C for details), compared with the model selection method by (Tang & Wiens 2021 which uses three offline policy evaluation (OPF)

---

Under review as a conference paper at ICLR 2022
PMS 8955
3.821
PNG-Th333%
1M5-71027%
TOE-Tha10%
373
341
340
24-
DPC
Figure 1: Model selection algorithms for offline DQN learning: (a) proposed pessimistic model selec-
tion (PMS); (b) weighted importance sampling (WIS) (Gottesman et al. 2018 (c) approximate model (AM) (Voloshin al. 2019;; (d) fitted Q evaluation (FQF) (Le al. 2019.. In this figure, the algorithms are trained and evaluated in a navigation task (E2) discussed Section and Appendix
estimates for validation. Specifically, based on the derived confidence interval of the OPE value for each candidate model, the final selected model by our PMS method is the one that has the largest lower confidence limit, which exactly has the largest true OPE value among all candidate models. In contrast, none of three OPE estimates used for model selection by (Tang & Wiens (202T) can identify the best model due to the inevitable overfitting issue during the validation procedure.
To close this section, we summarize the contributions of this work as follows:
We propose novel PMS framework, which targets finding the best policy from given candidate models (e.g., neural architecture, hyperparameters, etc) with offline data for DQN learning. Un- like many existing methods, our approach essentially does not involve additional hyperparameter tuning except for two interpretable parameters.
Leveraging asymptotic analysis in statistical inference, we provide uncertainty quantification on each candidate model, based on which our method can guarantee that the worst performance of finally selected model is the best among all candidate models. See Corollary [oor more details.
To address potential biases of candidate models in identifying the optimal policy, two refined approaches are proposed, one of which can be shown to have regret bounded by the smallest error bound among all candidate models under some technical conditions (See Corollary [ To the best of our knowledge, this is the first model-selection method in offline DRL with such a guarantee.
The numerical results demonstrate that the proposed PMS shows superior performance in different DQN benchmark environments.
2 RELATED WORK
Model Selection for Reinforcement Learning: Model selection has been studied in online decision-making environments (Fard & Pineau| 2010 Lee & Taylor 2014). Searching nearly opti- mal online model is a critical topic for online bandits problems with limited information feed-backs. For linear contextual bandits, Labbasi-Yadkoo et al. (20TT); Chu et al.] (20TT) are aiming to find the best worst-case bound when the optimal model class is given. For model-based reinforcement learning,PPcceee PPachhiaoo et al. (2020) introduces advantages of using noise augmented Markov Decision Processes (MDP) to archive competitive regret bound to select
individual model with con- straints for ensemble training. Recently, [ee et al. (202T) utilized an online algorithm to select a low-complexity model based on statistical test. However, most of the previous model selection approaches are focused on the online reinforcement learning setting. Very few works including al. (2020): (Su et al. (2020); Yang et al. 2020); Kuzborskij et al. (222 ; Tang & Wiens (2021), y; Xie & Jiang (2021) are focused on the offline setting. In partic- ular, (Su al. al.]| [0200] Yang al. al.] [0.00) Kuzzborkk et all 202T) focus on model selection for OPE problem. (Farahmand & 270pesvári[2011]]p X|eee & Jiang 2021) select the best model/policy based on minimizing the Bellman error, while the first approach requires an additional tuning and latter does not. (Paine et al., 2020 Tang & Wiens 202I)) proposed several criteria to perform model selection in OfiLL and mainly focused on numerical studies. In this work, we provide one of the first model selection approaches based on statistical inference for RL tasks with offline data collection.
Farahmand & Szepesvári (0TTT;
Offline- Policy Learning: Training a DRL agent with offline data collection often relies on batch- wise optimization. Caath--onstr deep Q-leanning (Fujimoto et al. 2019) (BCQ) is considered one OffRL benchmark that uses a generative model to minimize the distance of selected actions to the batch-wise data with perturbation model maximize its value function. Other popular OMRI.
2

---

Under review as a conference paper at ICLR 2022
2019), and random ensem- ble mixture (Agarwal et al. 2020) (REM) (as an optimistic perspective on large dataset), have also been studied in RL Unplugged (RLU) (Gulcehre et al. 2020) benchmark together with behavior cloning (Bain & Sammut 1995 Ross & Bagnell| 2010) (BC), DQN, and DQN with quantile regres- sion (Danney et aal]][211 8 (QR-DQN). RLU suggests naive approach based human experience for offline policy selection, which requires independent modification with shared domain expertise (e.g., Atari environments) for tuning each baseline. Meanwhile, how design a model selection algorithm for OffRL remains an open question. Motivated by the benefits and the challenges as mentioned earlier of the model selection for offline DRL, we aim to develop unified approach for model selection in offline DRL with theoretical guarantee and interpretable tuning parameters.
approaches, such behavior regularized actor-critic (BRAC) (Wu al.
3 BACKGROUND AND NOTATIONS
Consider a time-homogeneoue Markov decision process (MDP) characterized by a tuple M = (S, A. p. 7), where S is the state space, A is the action space, p is the transition kernel, i.e., p(s"|s, a) is the probability mass (density) of transiting to s' given current state-action (8.0) a), r is the reward function, i.e., E (R..S. 8, A, a) r(s, a) for & 2 O, and 0 S 7 1 is dis- count factor. For simplifying presentation, we assume A and S are both finite. But our method can also be applied in continuous cases. Under this MDP setting, it is sufficient consider sta- tionary Markovian policies for optimizing discounted sum of rewards (Puterman 1994). Denote as a stationary Markovian policy mapping from the state space S into a probability distribution over the action space. For example, T[aas)) denotes the probability of choosing action given the state value 8. One essential goal of RL is to learn an optimal policy that maximizes the value function. Define (*))) = - 7 [Re]So = s) and then the optimal policy is defined as 't E argmax, (V(T) _ (1 >- (s)s)((s), where v denotes some reference distribution function over S. In addition, we denote Q-function (s, - ''E" (RE' So s) for E S and E A. In this work, we consider the OffL setting. The observed data consist of N trajectories, corresponding {(St, A,, Re)3 For any i E {1,-... .n), data collected from the ith trajectory can be sum- marized by {(S..,, Ar,, R.,,, (1,,41))oscee where T denotes the termination time. We assume that the data are generated by some fixed stationary policy denoted by b.
N independent and identically distributed copies of
Among many RL algorithms, we focus on Q-learning type of methods. The foundation is the optimal Bellman equation given below.
Q' as)aa) = ElR, + max Q' ("(((1,,a') S. = =,A, a].
d'EAA
where Q" is called optimal Q-function, i.e., Q-function under n=" Among others, fitted q-iteration
(FQI) is one of the most popular RL algorithms (Ernst et al.] 2005). FQI leverages supervised learning techniques iteratively solve the optimal Bellman equation and shows competitive performance in OffRL.
To facilitate our model-selection algorithm, we introduce the discounted visitation probability, mo- tivated by the marginal importance sampling estimator in (Liu et al.) 2018)). For any t 2 0, let Pī (s, a) denote the t-step visitation probability Pr" s, = a) assuming the actions are se- lected according to π at time 1, 1.... ,t. We define the discounted visitation probability function as d' (s, a) (1 - 7) Σ > pp (s. a). To adjust the distribution from behavior policy to any target
policy n, we use the discounted probability ratio function defined as
d= (5) (S)
(s,
(2)
ICO (s)
where ps(, a) is the t-step visitation probability under the behavior policy b, i.e., (r. = A a). The ratio function (s, a) is always assumed well defined. The estimation of ratio function motivated by the observation that for every measurable function f defined over S x A.
E
" AL)((S,) A.) -Y Σ a(a' ( d')]
t=
a'EA
7) Bso
Σ a(a| So)},, a
(3)
3

---

Under review as a conference paper at ICLR 2022
based which several min-man estimation methods has been proposed such as (Liu et al. 2018
Nachum et al.) 2019| Uehara & Jiang 2019); We refer to (Uchara & Jiang 2019 Lemma 1) for
formal proot of equation
Finally, because our proposed model selection algorithm relies on an efficient evaluation of any target policy using batch data, we introduce three types of offline policy evaluation estimators the existing RL literature. The first type is called direct method via estimating Q-function, based on
the relationship that V(TT) -1)-)) 2.ES. π (s)s))((s, (s))))((s). The second type is motivated
by the importance sampling (Precup) 2000). Based on the definition of ratio function, we can see
(ITT) = 7 Ei=o (S,, A,)R,, from which plugin estimator can be constructed. The last type of OPE methods combines the first two types of methods and construct a so-called doubly robust estimator (Kallus & Uehara [019 Tang et al. (2020b). This estimator is motivated by the efficient influence function of VTT under transition transition-sampline sampling setting and the model that consists of the set of all observed data distributions given by arbitrarily varying the initial, transition, reward, and behavior policy distributions, subject to certain minimal regularity and identifiability conditions (Kallus & Uchara 2019). i.c.,
T Σ
w F. (St, AD)R +y
Σ
- (S, (S,,A))
7) Es (aals))Q (So. a)] V(T).
(4)
aEA
A nice property of doubly robust estimators is that as long either the Q-function (s, a) or the ratio function w wT (s, a) can be consistently estimated, the final estimator of (TT) is consistent
(Robins et al., 1994 Diang& & Li 2015 Kallus & Uehara 2019 Tang et al. 2020b). Furthermore,
doubly robust estimator based on (4) can achieve semiparametric efficincy under the conditions proposed by (Kallus & Ucharal 2019), , even if nuisance parameters are estimated via black box
models such as deep neural networks. Therefore such an estimator is particularly suitable under the framework of DRL. Our proposed algorithm will rely on this doubly robust type of OPE estimator.
4 PESSIMISTIC MODEL SELECTION (PMS) FOR BEST POLICY
In this section, we discuss our pessimistic model selection approach. For the ease of presentation, we focus on the framework of (deep) Q-learning, where policy optimization is performed via estimating the optimal Q-function. While this covers a wide range of state-of-the-art RI. algorithms such as FQI (Ernst et al. 2005), DQN (Mnih et al. (203)) and QR-DQN (Dabney et al.) 2018), we remark that our method is not restricted to this class of algorithms.
Suppose we have total number of L candidate models for policy optimization, where each candidate model will output an estimated policy, say मं for 1 S S L. Our goal to select the best policy among L policies during training procedure. Note that these L models can be different deep neural network architectures, hyper-parameteee and various classes of functions for approximating the optimal Q-function or policy class, etc.
4.1 DIFFICUTIESS AND CHALLENGES
Given a candidate among I models, we can apply for example FQI using the whole batch data Dn to learn an estimate of Q as Qi and an estimated optimal policy कn defined as (als)) e (rimax.eAQQ(s,,. a), for every E S. In order to select the final policy, one may greedy approach to choose some such that goal is to maximize (TT). However, using this criterion will lead to over-fittigg Specifically, due to the distributional mismatch between the behavior policy and target policies, which is regarded as a fundamental challenge in OffRL (Levine et al.) 2020). we may easily overestimate Q-function, especially when some state-action pairs are not sufficiently visited in the batch data. This issue becomes more serious when we apply max-operate during our policy optimization procedure. Such observations have already been noticed in recent works, such as (Kumar al. 2019: 2020 Paine al. 2020 Yu et al.] 2020 Tang & Wiens (222 et al.) 202T). Therefore, it may be inappropriate use this criterion for selecting the best policy among T. models.
naive Deanaax,ssslEeess. at(aas) a)), as our
4

---

Under review as a conference paper at ICLR 2022
One may also use cross- voossvalidation procedure to address the issue of over-fitting or overestimating Q-function for model selection. For example, one can use OPE approaches on the validate dataset to evaluate the performance of estimated policies from the training data set (see Taag & Wiens (202T) for more details). However, since there is no ground truth for the value function of any policies, the OPE procedure on the validation dataset cannot avoid involving additional tuning on hyperparameters. Therefore such a procedure may still incur large variability due to the over- fitting issue. In addition, arbitrarily splitting the dataset for cross-validation and ignoring the Markov dependent structure will cause additional errors, which should be seriously taken care of.
4.2 SEQUENTIAL MODEL SELECTION
In the following, we propose pessimistic model selection algorithm for finding an optimal policy among I candidate models. Our goal is to develop an approach to estimate the value function under each candidate model during our policy optimization procedure with theoretical guarantee. The proposed algorithm is motivated by recent development in statistical inference of sequential decision making (Luedtke& & Van Der Laan 2016 Shi et al., 2020j. The idea is to first estimate optimal Q-function Q", optimal policy x' and the resulting ratio function based and evaluate the performance of the estimated policy on the next chunk of data using previously estimated nuisance functions. Then we combine the first two chunks of data, perform the same estimation procedure and evaluation on the next chunk of data. The framework of MDP provides a nature way of splitting the data.
chunk of data,
Specifically, denote the index of our batch dataset Dn as Jo - divide Jg into O number of non-overlapping subsets, denoted by J1.-.. Jo and the corresponding data subsets are denoted by D..... Do. Without loss of generality, we assume these data subsets have equal size. We require that for any S 01 O2 S O, any (i1, t1) E Jo, and (1g, t2) e Jog. either i2 + in or l < 12. For 1 < S O, denote the aggregate chunks of data as D.- = { Ri,e, S1,e+1), (i, E Jo = J1 U...UUU. U...
We
We focus on FQI algorithm for illustrative purpose and it should be noticed that our algorithm can be applied to other RL algorithms. Starting from the first chunk of our batch data, at the o-th step (o 1, 1,...
1), for each candidate model I 1, ... Lt, we apply FQI on D. to compute Q as an estimate of optimal Q-function and obtain (0) correspondingly such that (() (als) E rrgmax.e.A0
Q111)) (s.a) for every E S. Additionally, we compute an estimate of ratio function w+ using D. by many existing algorithms such Nachum et al. (211)). Denote the resulting estimator as @=(")," : The purpose of estimating this ratio function is to improve the efficiency and robustness of our value function estimation for each candidate model. Then compute the
estimated value function of 2[() on Do+1 as
DD.11(#1 (#(0)) = (1 Esp Σ
() (So. (S))
(5)
ADEA
E 1 O
(S,
Σ
#
(6)
where ED.41 denotes the empirical average over the (o+ 1) chunk of dataset and (S, A, R, S') is
one transition tuple in Do+1- While one can aggregate Di (*( for S o S (0-1) to evaluate the performance of L models, the uncertainty of these estimates due the finite sample estima- tion should not be ignored. Therefore, in the following, we derive an uncertainty quantification of our estimated value function for each candidate model, for performing model selection. Based on
equation 4 (conditioning D.), the variance of VD... (2 is
o² (+((0)) = (x "(S,A)(H - Σ a/00 (dis't))
(7)
a'EAA
where (S, A, S') is a transition tuple with (S, A) follows some stationary distribution. See Assump- tion Correspondingly we have an estimate defined as
(+(")) Er.., (10
+
Σ a'EA
*1
(d'ls") (S'.a') (S,A)), (8)
5

---

Under review as a conference paper at ICLR 2022
The estimation procedure stops once have used all our offline data and denote the final estimated policy as ft for each 1.... L. Notice that क = # Finally, we compute the weighted average of all the intermediate value functions as our final evaluation of the estimated policy कt, i.e.
VD (A
((t) =
(9)
(
(
(5)
In Section 5 show that under some technical conditions, the following asymptotic result holds:
VnT(O- - 11/0 (V(*)) V(Ri)
(N,, 1),
(10)
8(1)
refers to weak convergence when either or T goes infinity, and No, 1) refers to the standard normal distribution. Based on the asymp- totic result in (TO), we can construct confidence interval for the value function of each policy in.
where a1) = (0- 1) 0-1
p=1 (
Given confidence level a, for each I, we compute (1) (ft) 2a/2 / /O/nI(O (1), where Ea/2 is (1 -quantile of the standard normal distribution. Our final selected one is EE argmax (1).
The use of (1) is motivated by the recent proposed pessimistic idea to address the overestimation issue of value (or Q) function in the OffLL setting. See Kumar et al. (2019 2020); Jin et al. (2021);
Xie et a (22T): Uehara & Sun (20TT); Zanette al. (2021) for details. The final output of our algorithm गyy and an outline of the proposed algorithm can be found in Algorithm I As can see, our algorithm is nearly tuning free, which provides great flexibility in real-world applications. The only adjustable parameters is O and a. The size of O balances the computational cost and the fimite-sample accuracy of evaluating each candidate model. In specific, we can indeed show that the variance of the estimated value function by our algorithm can achieve the cemi-parametric efficiency bound, which is best one can hope for. So in the asymptotic sense, the effect of O is negligible. In the finite-sample setting, we believe the performance will be discounted by factor vo - 1/-1/O. Therefore, if O is large enough, Vo - 1/O will have mere effect on the performance. See Theorem [ However, using large O will result in a large computational cost. As a sacrifice for the nearly tuning free algorithm,
need apply OffL algorithms O times for each candidate model. The parameter a determines how worst the performance of each policy we should use to evaluate each policy. See Corollary [ for more insights.
Algorithm 1: Pessimistic Model Selection (PMS) for OffRI.
Input: Dataset D, and L candidate models for estimating optimal Q-function and policy; We divide D, into non-overlapping subsets denoted by Di,,.. Do. We require that for any 01 02 < O, any (61,4) Jon and (iz, (2) Jog either i2 or S l2-
for | E L do
for o = 1 O do
For I E L models, construct the optimal Q and + using D. data subset.
Compute 0 using D, by Nachum et al. 2019 and min-max solver for (
Compute Dr..., (o) and 2 (I) using Do+1 given in (5) and ( respectively.
For 1-th model, we compute (()) (*t) - 2. - 1)/Oa where (*t) and
(1) are given in ( and 10 respectively.
Pick = arg max; U (1) as the selected model and run the algorithm on full dataset to obtain क
Return nt.
THEORETICAL RESULTS
In this section, we justify our asymptotic result given in 10 We use Op to denote the stochastic boundedness. Before that, we make several technical assumptions:
Assumption The stochastic process (At, St Sex is stationary with stationary distribution Pre: Assumption For every S S L and 1 < o S O. we have E ((+ -V((=VV(())))))))))> - for some constant Co and > 1/2.
O)

---

Under review as a conference paper at ICLR 2022
Assumption 1 is standard in the existing literature such (Kallus & Uehara 2019). is key to our developed asymptotic results developed. This assumption essentially states that all candidate models are good enough that eventually their value functions will converge to that of the optimal one. This implies that there is no asymptotic bias in identifying the optimal policy. While this is reasonable thanks to the capability of deep neutral networks, which has demonstrated their empirical success in many RL applications, such an assumption could still be strong. In Section 6 we try to relax this assumption and provide two remedies for addressing possibly biased estimated policies. In addition, Assumption [ also requires that the convergence rates of value functions under estimated policies are fast enough. This has been shown hold under the margin condition on '², see e.g., (Hu et al. 2021) for more details.
Assumption
Assumption 3 For every < S L and 1 1 o o 10-0-- - suppose E(s,A)~pp O A) Q41 A)]² Op((nT/O) -2x1 for some constant Ky 2 O. In addition, ②100 is uniformly bounded almost surely.
Assumption 4 For every 1 1S15 L and S o S 0 suppose B(S,A)~p= 10a1 "(S, A)
w A)] Op(nnI/O)~
for some constant R2 2 0. In addition, both w and " are uniformly bounded above and below away from 0 almost surely.
Assumption For every 1 S S L, and S o S 0 - (4 and o2 (#)) are bounded above and below from 0 almost surely.
Assumptions A impose high-level conditions on two nuisance functions. Our theoretical results only require K + > 1/2, which is mild assumption. For example, if considered parametric models for both Q-function and ratio function, then K = K2 1/2. If considered nonparametric models for these two nuisance functions such as deep neural networks, then 1/4 < K1, < 1/2 can be obtained under some regularity conditions. See[Fan et al. (2020) and [iao et al. (2020); Uehara iet al. (2021) for the convergence rates of Q-function and ratio function by non non-parametric models respectively. In addition, Assumption 5 is a mild assumption, mainly for theoretical justification. Then we have the following main theorem as a foundation of our proposed algorithm.
Theorem Under Assumptions
we have
(ViT(O- (v())) - V(t))) No(
(11)
Theorem provides an uncertainty quantification of each candidate model used in policy optimiza-
tion. Such uncertainty quantification essential in OffRL as data are often limited. We highlight
the importance of such results in Appendix A A consequent result following Theorem I validates
the proposed Algorithm
Corollary inf Pr(V(#t) 2 max V(+t) - 22,, (
2 - Lo under
+50
Assumptions
As can be seen clearly from Corollary and the proposed PMS, with a large probability (by letting small), we consider the worst performance of each candidate model Ωt in the sense of the lower confidence limit of the value function, and then select the best one among all models.
TWO REFINED APPROACHES
In this section, we relax Assumption 2 by allowing possibly non-negligible bias in estimating the optimal policy and introduce two refined approaches for addressing this issue. Instead of imposing Assumption] 2
we make an alternative assumption below.
Assumption For every 1 S I S L, there exists B(I) such that maxigos(o-1) (V(4
V(=*)) < B() almost surely.
Assumption is very mild assumption. It essentially states that the biases for all our intermediate value function estimates are bounded by some constant, which is much weaker than Assumption 2 In this case, the asymptotic results in (11) may not hold in general. Correspondingly, we have the following result.
Theorem Under Assumptions 7 3 6
for every 1 S l L, the following inequality holds.
lim inf Pr (V(==")
+ BBI)) 2 =1-a.
(12)
7

---

Under review as a conference paper at ICLR 2022
Motivated by Lepski's principle (Lepski & Spokoiny 1997) from nonparametric statistics and (Su et al. 2020) studying the model selection of OPE, we consider the following refined model-selection procedure to find the best policy. We first rank L candidate models in non non-increasing order based on the value of δ (1). i.e., for S i < j I I,, (i) 2 o()). Then for i-th model, we construct an interval as (() = ( - / O0 -
V(+) + 222/(22) (0///((--)))////// - Finally the optimal model/policy we choose is मं such that i max(i 1 isis (,nsss + 0). To show this procedure is valid, we need to make one additional assumption.
Assumption There exists Ç < 1 such that for 1 S i i L. B(i)= < B(i + 1) and (ā(i) v
a + 1) S (()) almost surely.
This assumption borrowed from SSu et al. (2020). It typically assumes that after sorting our model based on ol)) the bias of estimated policy is monotonically increasing and the standard deviation is monotonically deceasing but not too quickly. This is commonly seen when all candidate estimators exhibit some bias-variance trade-off phenomena. Define the following event
ع ( ((T") s 6(1 + (")+ min {B(i) + 20/(22) VO/nT(O- (1)(
(13)
ISiSL
Then we have the following theoretical guarantee for our refined procedure.
Corollary 2 Under Assumptions 7 3
7 have lim inf Pr(E) 2
If we further assume
-000000 that for any 8 > O, with probability at least 1 - 5, for every 1 S i S I, (V(a.) - ((+))) s
c( log( (1) VNT for some constant c(0), then iim Pr(E) > 1 - - 6, where
some
nf
E V(tt)) V(+')I s 3(1+ min (B + (c(6) log(L) + = 1)0( (14) 1SISL
The additional assumption (i.e., the high probability bound) in Corollary 2 can be shown to hold by the empirical process theory under some technical conditions (Van de Geer 2000). Hence Corollary 2 provides strong guarantee that the regret of the final selected policy is bounded by the smallest error bound among all I. candidate policies. Note that Assumption 3 imposed here could be strong.
Another refined approach: Notice that the above refined approach indeed focuses on OPE esti- mates to select the best policy with regret warranty. The rough motivation behind is to find a policy that has the smallest estimation error to the optimal one. However, such procedure may not directly match the goal of maximizing the value function in OffRL. To relieve this issue, we can alternatively
choose the final policy as â such that i = argmax, 1 ((t.) 22a/2 /nt/:///TO//-o-// 1))) where the argmax is taken over models. This approach can be viewed as a combination of PMS and the above refined approach. By adopting this approach, candidate models with large biases are firtly removed by the truncation on i. Then, we use the idea of PMS select the best model having the best worst performance among the remaining candidates. Unfortumately, we do not have theoretical guarantee for this combined approach.
7 EXPERIMENTAL RESULTS
We select six DQN environments (E, to Eg) from open-source benchmarks (Brockman et al. 2016 Juliani et al. 2018 g to conduct numerical experiments. as shown in Fig.5500 Appendix C These tasks of deployed enviromments cover different domains that include tabular learning (Fig a)); automatic navigation in a geometry environment with a physical ray-tracker (Fig b)): Atari digital gaming (Fig((o) and (d)), and continuous control (Fig (je) and (f)). We provide detailed task description and targeted reward for each environment in Appendix C
Experiment setups. To evaluate the performance of PMS with DQN models in offRL. we choose different neural network architectures under five competitive DRL, algorithms including DON by Mnih et al. 2013 2015), BCQ by (Fujimoto et al. 2201).. BC by (Bain & Sammut, 1995| Ross & Bagnell 2010), BRAC by (Wu et al. 2019) from RLU benchmarks, and REM by (Agarwal et al,, 2020). Within each architecture, 70 candidate models are created by assigning different hyperpa- rameters and training setups. See Appendix C.I for details. We then conduct performance evaluation of different OffRL model selection methods these generated candidate models.
Evaluation procedure. We utilize validation scores from OPE for each model selection algorithm, which picks the best (or a good) policy from the candidate set of size T. based on its own criterion.
8

---

Under review as a conference paper at ICLR 2022
selected policy
06
Regret
04
02
00
Figure Sensitivity analysis different training data size. PMS attains the best performance and has the least sensitivity.
Figure 2: Box plots of model selection performance
from offline learning in each DRL algorithms for E,.
Regret is used the evaluation metric for each candidate. The regret for model 1 is defined as V (TT-) - V (ft), where / arg =1. V (TV.) corresponds to the candidate policy with the best OPE validation performance. In our implementation, we treat Ti. T², the oracle but unknown best possible policy. A small regret is desirable after model selection. Note the optimal regret is not zero since we can only use data obtain # instead of æı for each model. We provide additional top-k regret and precision results from Figure 6 to 16 Appendix [
Performance comparison. As highlighted in Fig. iin the introduction, we report estimated OPE values by different model selection approaches, i.e. PMS and three methods by (Tang & Wiens, 2021), versus the true OPE values. In this experiment, we consider 70 DQN models under the above mentioned five DRL algorithms, i.c., 14 models are considered for each architecture. We use fewer models for each DRL algorithm mainly for clear presentation. By using the confidence interval constructed by PMS procedure, while the other three methods fail. To further investigate the performance of PMS, we implement model selection among 70 models within each DRL algorithm separately. Fig. 2 shows the box plots of averaged regret over six environments after OPE per neural network architecture. Each subfigure contains results from one particular DRL algorithm with different hyperparameters or training setups. The left box plot refers to the regrets of all 70 models and the right one represents the regrets of top 10% models selected by the proposed PMS method. Note that the right box plot a subset of the left one. The results show that our proposed PMS successfully helps to select models with the best policies and improve the average regret by significant margin. particular, PMS- REM-based models attain the lowest regrets, due to the benefit from its ensemble process. Detailed results for each environment is given in Appendix C
method is able correctly select the top models,
Sensitivity analysis. Fig- compares different selection algorithms with varying training data size. PMS outperforms others across all scales, and larger number of episodes gives smaller variation and lower sensitivity.
Regret
02
PMS algorithm with refinements. We replicate experiments on in the offline navigation task in Eg (Banana Collector) for 30 times and report regrets of top 10% models selected by PMS and two refinements in Fig. As we can see, while the overall per- formances of the proposed three model selection methods are sim- ilar, two refined approaches have better regrets than PMS in terms of median, demonstrating their promising performances in identifying the best model. OPE results have been also evaluated also in DRL tasks with E, and Eg to Eg. where the refinement algorithms (PMS R R1/R2) have only a small relative + 0.423 performance difference compared to its original PMS setups.
PMs
PMS-A1
PM5.
Figure PMS and its refine- ments (R1/R2).
CONCLUSION
We propose new theory-driven model selection framework (PMS) for offline deep reinforcement learning based on statistical inference. The proposed pessimistic mechanism is warrants that the worst performance of the selected model is the best among all candidate models. Two refined ap- proaches are further proposed to address the biases of DRL models. Extensive experimental results on six DQN environments with various network architectures and training hyperparameters demon- strate that our proposed PMS method consistently yields improved model selection performance over existing basclines. The results suggest the effectiveness of PMS as a powerful tool toward automating model selection in offline DRI.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Yasin Yadkor,, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems, 24:23 12-2320, 2011.
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference Machine Learning, pp. 104-114, PMLR, 2020.
Michael Bain and Claude Sammut. A framework for behavioural cloning. In Machine Intelligence 15. pp. 103-129, 1995.
Gabriel Barth-Maron, Matthew W Hoffman, David Budden, Will Dabney, Dan Horgan, TB Dhruva, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributed distributional deterministic policy gradients. International Conference on Learning Representations, 2018.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv: 01540, 2016.
Wei Chu, Lihong Li, Lev Reyzin, and Robert Schapire. Contextual bandits with linear payoff func- tions. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pp. 208-214. JMLR Workshop and Conference Proceedings, 2011.
Will Dabney, Mark Rowland, Mare Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In Proceedings of the AAAI Conference Artificial Intelli- gence, volume 32, 2018
Damien Ernst, Pierre Geurts, Louis Wehenkel, and L. Littman. Tree-based batch mode reinforce- ment learning. Journal of Machine Learning Research, 6:503-556, 2005.
Jianqing Fan, Zhaoran Wang. Yuchen Xic, and Zhuoran Yang. theoretical analysis of deep q- learning. In Learming for Dynamics and Control, pp. 486-489, PMLR, 2020.
Amir-massoud Farahmand and Csaba Szepesvári. Model selection in reinforcement learning. Ma-
chine learning, 85(3):299-332, 2011.
M Fard and Joelle Pineau. Pac-bayesian model selection for reinforcement learning. Advances in Neural Information Processing Systems, 23:1624-1633 2010.
David A Freedman. On tail probabilities for martingales. the Annals of Probability, pp. 100-118, 1975.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pp. 2052-2062 PMLR, 2019.
Omer Gottesman, Fredrik Johansson, Joshua Meier, Jack Dent, Donghun Lee, Srivatsan Srinivasan, Linying Zhang, Yi Ding, David Wihl, Xuefeng Peng, et al. Evaluating reinforcement learning algorithms in observational health settings. arXiv preprint arXiv: /2298, 2018.
Caglar Gulcehre, Ziyu Wang, Alexander Novikov, Tom Le Paine, Sergio Gomez Colmenarejo, Kon- rad Zolna, Rishabh Agarwal, Josh Merel, Daniel Mankowitz, Cosmin Paduraru, et al. RI un- plugged: Benchmarks for offline reinforcement learning. arXiv e-prints, pp. arXiv-2006, 2020.
Hannes Hase, Mohammad Farid Azampour, Maria Tirindelli, Magdalini Paschali, Walter Simson, Emad Fatemizadeh, and Nassir Navab. Ultrasound-guide robotic navigation with deep reinforce- ment learning. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5534-5541. IEEE, 2020.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018.
Yichun Hu, Nathan Kallus, and Masatoshi Uchara. Fast rates for the regret of offline reinforcement learning. arXiv preprint arXiv:2 2021.
10

---

Under review as a conference paper at ICLR 2022
Natasha Jaques, Judy Hanwen Shen, Asma Ghandeharioun, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Human-centric dialog training via offline reinforcement learning. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 3985-4003, 2020.
Nan Jiang and Lihong Li. Doubly robust off- aff-policy value evaluation for reinforcement learning. arXiv preprint arXiv:1511.03722. 2015.
Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline ri? In International Conference Machine Learning, pp. 5084-5096. PMLR, 2021.
Arthur Juliani, AmcenttPPiere Berges, Ervin Teng, Andrew Cohen, Jonathan Harper, Chris Elion, Chris Goy, Yuan Gao, Hunter Henry, Marwan Mattar, et al. Unity: A general platform for intelli- gent agents. arXiv preprint
0262972 2018.
Gregory Kahn, Adam Villaflor, Bosen Ding, Pieter Abbeel, and Sergey Levine. Seff-supervised deep reinforcement learning with generalized computation graphs for robot navigation. In 2018 8 IEEE International Conference on Robotics and Automation (ICRA), pp. 5129-5136. IEEE, 2018.
Nathan Kallus and Masatoshi Uehara. Efficiently breaking the curse of horizon: Double reinforce- ment learning in infinite-horizon processes. arXiv preprint arXiv: (w0.999999999 2019.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. In NeurIPS, 2020.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Sys- tems, pp. 17784-11794, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2006. 04779, 2020.
Ilja Kuzborskij. Claire Vernade, Andras Gyorgy, and Csaba Szepesvári. Confident off-policy eval- uation and selection through normanordl importance weighting. International Conference on Artificial Intelligence and Statistics, pp. 640-648. PMLR, 2021.
Hoang Le, Cameron Voloshin, and Yisong Yue. Batch policy learning under constraints. In Inter- national Conference on Machine Learning, pp. 3703-3712, 2019.
Jason D Lee and Jonathan E Taylor. Exact post model selection inference for marginal screening. In Proceedings of the 27th International Conference on Neural Information Processing Systems- Volume I, pp. 136-144, 2014.
Jonathan Lee, Aldo Pacchiano. Vidya Muthukumar, Weihao Kong, and Emma Brunskill. Online model selection for reinforcement learning with function approximation. In International Con- ference Artificial Intelligence and Statistics, pp. 3340-3348. PMLR, 2021.
Oleg V Lepski and Vladimir G Spokoiny. Optimal pointwise adaptive methods in nonparametric estimation. The Annals of Statistics, pp. 2512-2546, 1997.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01663 2020.
Peng Liao, Zhengling Qi, and Susan Murphy. Batch policy learning in average reward markov decision processes. arXiv preprint arXiv:2007.11771 2020.
Qiang Liu, Lihong Li. Ziyang Tang, and Dengyong Zhou. Breaking the curse of horizon: Infinite- horizon off-policy estimation. In Advances in Neural Information Processing Systems, pp. 5356- 5366, 2018
Alexander R Luedtke and Mark Van Der Laan. Statistical inference for the mean outcome under a possibly non-unique optimal treatment strategy, Annals of statistics, 44(2):713, 2016.
Peter Mathé. The lepskii principle revisited, Inverse problems, 22(3):L111 2006.

---

Under review as a conference paper at ICLR 2022
Donald L McLeish. Dependent central limit theorems and invariance principles. the Annals of Probability, 2(4))620 628, 1974.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:13/2.5002 2013.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning, nature, 51 518(7540):529-4 -33,, 2015.
Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections. In Advances in Neural Information Processing Systems, pp.
2019.
Aldo Pacchiano, Philip Ball, Jack Parker-Holder, Kızysztof Choromanski, and Stephen Roberts. On optimism in model-based reinforcement learning. arXiv preprint arXiv:2006. / 1911, 2020.
Tom Le Paine, Cosmin Paduraru, Andrea Michi, Caglar Gulcehre, Konrad Zolna. Alexander Novikov, Ziyu Wang, and Nando de Freitas. Hyperparameter selection for offline reinforcement learning. arXiv preprint urXiv:2007.09055 2020.
Doina Precup. Eligibility traces for off-policy policy evaluation. Computer Science Department Faculty Publication Seriex, pp. 80, 2000.
Martin L Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, Inc., 1994.
James M. Robins, Andrea Rotnitzky, and Lue Ping Zhao. Estimation of regression coefficients when some regressors are not always observed. Journal of the American Statistical Associa- tion, 894427):446866, 1994. ISSN 01621459. URL htth::tt///m jstor wot/ooraglleee... 2290910
Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 661-668. JMLR Workshop and Conference Proceedings, 2010.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint 15/1055 2015.
Chengchun Shi, Sheng Zhang, Wenbin Lu, and Rui Song. Statistical inference of the value function for reinforcement learning infinite horizon settings. arXiv preprint a:X/v:2001.04515. 2020.
Noah Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller, Keep doing what worked: Behavior modelling priors for offline reinforcement learning. International Conference on Learning Representations, 2019.
Satinder P Singh and Richard S Sutton. Reinforcement learning with replacing eligibility traces. Machine learning, 22(1):123-155 1996.
Yi Su, Pavithra Srinath, and Akshay Krishnamurthy. Adaptive estimator selection for off-policy evaluation. In International Conference on Machine Learning, pp. 9196-9205. PMLR, 2020.
Shengpu Tang and Jenna Wiens. Model selection for offline reinforcement learning: Practical con- siderations for healthcare settings. arXiv preprint vrXiv:2107111003 2021.
Shengpu Tang, Aditya Modi, Michael Sjoding, and Jenna Wiens. Clinician-in-the-lopp decision making: Reinforcement learning with near-optimal set-valued policies. In International Confer- ence on Machine Learning, pp. 9387-9396. PMLR, 2020a.
Ziyang Tang, Yihao Feng, Lihong Li, Dengyong Zhou, and Qiang Liu. Doubly robust bias reduction in infinite horizon off-policy estimation. In International Conference on Learning Representa- fions, 2020b.
12

---

Under review as a conference paper at ICLR 2022
Masatoshi Uehara and Nan Jiang. Minimax weight and q-function learning for off- aff-policy evaluation. arXiv preprint arXiv: 19I..v:910..200 1289,, 2019.
Masatoshi Uehara and Wen Sun. Pessimistic model-based offline rl: Pac bounds and posterior
sampling under partial coverage. arXiv preprint arXiv:2107.06226, 2021.
Masatoshi Uehara, Masaaki Imaizumi, Nan Jiang, Nathan Kallus, Wen Sun, and Tengyang Xie. Finite sample analysis of minimax offline reinforcement learning: Completeness, fast rates and first-order efficiency. arXiv preprint arXiv:2102.0298/ 2021.
Sara Van de Geer. Empirical Processes in M-estimation, volume Cambridge university press, 2000.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
Cameron Voloshin, Hoang M Le, Nan Jiang, and Yisong Yue. Empirical study of off off-policy policy evaluation for reinforcement learning. arXiv preprint arXiv: NXXiv:911.0065:: 2019.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv: 1911.11911.113000 2019.
Tengyang Xie and Nan Jiang. Batch value-function approximation with only realizability. In Inter- national Conference on Machine Learning, pp. 11404-11413, PMLR, 2021.
Tengyang Xie, Ching-An An Cheng, Nan Jiang, Paul Mineiro. and Alekh Agarwal. Bellman-coosistent pessimism for offline reinforcement learning. arXiv preprint arXiv:2 06926, 2021.
Chao-Han Huck Yang. Hung. Te Danny, Ouyang, and Pin-Yo Yu Chen. Causal inference q-network: Toward resilient reinforcement learning. arXiv preprint arXiv:1102.09677, 2021.
Mengjiao Yang. Bo Dai, Ofir Nachum, George Tucker, and Dale Schuurmans. Offline policy selec- tion under uncertainty. arXiv preprint arXiv:2012.06919, 2020.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. arXiv preprint 2rXiv:2005.13239 2020.
Andrea Zanette, Martin J Wainwright, and Emma Brunskill. Provable benefits of actor-critic meth- ods for offline reinforcement learning. arXiv preprint arXiv:2108.05 08812, 2021.
13

---

Under review as a conference paper at ICLR 2022
A COMMENTS ON ASYMPTOTIC RESULTS
We remark here that all theoretical justification in this paper based asymptoties. It might be possible to investigate finite sample regimes when one has an exact confidence interval instead of II or a non-anynmpaa non-asymptotic bound. However, having an exact confidence interval might require some model specification of the value function, and using non-asymptotic bounds might require additional tuning steps (e.g., constants many concentration inequalities), which beyond the scope of this paper. In addition, as seen from our empirical evaluations below, with relatively large sample size, the proposed model selection approach performs well.
B TECHNICAL PROOFS
The notation E(N) S ((N) (resp. E(N) 2 ((N)) means that there exists a sufficiently large (resp. small) constant c1 > 0 (resp. 02 0) such that {(N) S c1 (resp. ((N) 2 c20(N)) for some sequences ((N) and E(N) related to N. In the following proofs, N often refers to some quantity related and T. Lemma and
Notations:
and its proof: Let denotes some index of our batch data Da. Define
Σ ( And)):
6(J.Q TH ,TTH T)
(S,,, Bast +7 (G.))J
where |] is the cardinality of the index set J, e.g., JJl the following Lemma [ as an intermediate result to Theorem
nf for every S o O. Then have
Lemma Under Assumptions 0 and for every 1 s 1 S L and 1 5 0 -, the following
asymptotic equivalence holds.
nT Vol DD... (+))) - -
nT Vo HJ Qa w a(1)) + (1))..
(15)
where (1( refers to a quantity that converges to 0 as n or goes to infinity.
The proof is similar to that of Theorem 7 in Kallus & Uchara 2019 First, notice that
nT
VD-. (5(0)) -
nT
[o(J, @7t1l)) 100 " =1(0))
Φ[J, 25% ,w*!" ((0))
+ (1 Esone Σ (0) (alSn) (S., a)) - (1-7)Bson))
Σ (0) (a) (So,
aed
aed
+ nT
o 6JJ, Q
w ((0))
Then it suffices to show the term the first bracket converges to 0 faster than VnT. Notice that
( (0 ( ,(J 2 " :0 (0))
+ (1 -77)B Σ a (al))) (0) (So, a)) - (1 - B5))
Σ #| (alSo)) (So.
aed
acd
=E Eg + Es,
14

---

Under review as a conference paper at ICLR 2022
where
E1
0
Σ
(SE,,
" ((S,,, -Qth (See, Avd)
nT
(6,1)
+y Σ
(0)
(alSix+1) (S)+
#
weth * A (Sta, Ai.) - Qa: (Sco, Ai,))
E2
nT
(1
2/0 (a(S
(S).41,0)) Q4 - (St,41,)))),
+y
and
Es
Σ
(w v (Si A.,)) w (SSu,
(S (S,). And Q²: (St.s A1,))
nT
(1,0)EJ+1
(c()
(8) - Q (S),+1,))))
Next, we bound each of the above three terms. For term E,, can be seen that
[[E 0.
In addition, by Assumptions and we can show
Var F. ([[Va((F11Jo))) 2 N (nT/O) 2n2
where the inequality based on that each item in Ez is uncorrelated with others. Then by Markov's inequality, we can show
E1 Op(( 0 11/2-x2).
nT
Similarly, we can show
E Op((
11122-11)
n/
For term (Fg), by Cauchy Schwarz inequality and similar arguments as before, we can show
Eal = O nT
((m2tm1)).
Therefore, as long as (K2 + k1) > 1/2, we have E₁ + Eg + Es = olVO/nI)) which concludes our proof.
Proof Theorem We aim to show that
VnT(O - 1)/0 (i(#t)) - v(tt))
((0,
o(1)
It can be seen that
VnT(O 1)/0 (D(m)) v(
0
V (+)(0) V(Ti)
nT
o(1)
O 1)
00+1 (2(00)
DD (0() V(+ (0))
O(0 -1)
0o+1 (4(0))
nt
((+(()) - V(trs)
1 0(0-1)
(071(1(1)) (0))
15

---

Under review as a conference paper at ICLR 2022
Define
,.,,), m) Φ(J ,,,"
1
wh (Ste Ar)
Bis +
Σ (Q0 Q" (Stee (Ax):AA)))
J
(F,))J)
a'EA
where |J] is the cardinality of the index set J, i.e., |J = nT
Then by Lemma | we show that
0.
nT
VD..I (#1 V(2((0)
nT Φ(Jo+1, :Nott1.00 Qa .. #1())
+ oz(())
(16)
0o+1(At) (0)
00+1(A1
we can show that
5 (5t (2(0))
(0-5(0-1))
max
10+1 (5) 10))
- = Op()),
which will be shown later, then by Slutsky theorem, we can show that
VD (2/10) V(
nT
010-1 1)
5 0ot1 (5(())
0(J+1108 _w0+( a(0))
nT
top(1).
010 -1
Jot1 (#(
For (I), that
(1)
v nn
Σ
t Act)(Ba
(17)
(1.0)
(0 # (a' (d (St))1,))) - Q24( (Si, oo
(18)
d'EA
By the sequential structure of our proposed algorithm, (7) forms mean zero martingale. Then we use Corollary 2.8 of (McLeish, 1974) to show its asymptotic distribution. First of all, by the uniformly bounded assumption on Q-function, ratio function and the variance, we can show that
max max th P (Ri) +y
Σ "ं (d')S (S))+1,0))-
nT - 1) (0-5(0-1) (1.1)
a'Ed
Q4i
(Si, (00) =1((1)
Next, we aim to show that
0-1
Σ
Σ
(0)
(19)
nT o=l
1)
(1.1)..
+y Σ
( (t.)S
(Sa,41, Q+1 An,)P - = =
n'EA
Notice that the left hand side of the above is bounded above by
0 max nT
Σ
(w T (ix)).A.))Ri:
(20)
1505(0-11
+y Σ
(d'
- - And)
-
(21)
d'EA
16

---

Under review as a conference paper at ICLR 2022
Because, for each 1 1 10
Σ
tw
(22)
nT
(1.0)eJo11
+7
Σ
को
Ar,))} (S (a' ( (S((41,)) - (Sht,
- Bl( "0 "SS, A)R
(23)
d'EA
o (d'is") (5) (5.A)//100
(24)
a'E.A
forms mean zero martingale, we apply Freedman's ' inequality in (Freedman 1975) with Assump- tions to show it is bounded by Oply nrr) Applying union bound shows 19 is op(1) and
furthermore consistency of (ft) in (16) holds. Then we apply the martingale central limit theorem to show
(Jo+1, _0 ",""" _.(00)) ,,0"",
nT 0(0-1)
NOO, 1).
( (0))
The remaining is to show
nf 0(0 1)
V(*lo0) V(tt))
Zo+1 (5 (o)
asymptotically negligible. Consider
E ((#(()) v(tt))
(25)
E v( V(TT)
+ E(V(At) v(T
(26)
< v(+ - -V(T))
+ E/V(+)) v(Ti)l
(27)
$ (nTo) + (nT)""
(28)
where we use Assumption
2 for the last inequality. Summarizing together, show that
nT
0-1
E
Σ
((+(()) V(+i)
0
1)
0=1
nT
0-1
nT(O - (nT))
(nTo) -K0* +
0 - 1)
nTO²
0-1
nT(O - (nT))
Σ
(nIT)
0(0 - o=1
=o(1),
where we obtain the second inequality by that Eo-1 0 < x S 01-k In the last
inequality, we use x > 1 in Assumption Then Markov inequality gives that
nT vo(o-1) -1] 1)
Σ ( M(tt)) o1))
Moreover, by Assumption that inf15oso-100 further show that
8ot1 (5((0) 2 c for some constant c > 0, we can
nT 0 - 1)
((* V((tt)) o1)
dat1 (2)
which completes our proof.
17

---

Under review as a conference paper at ICLR 2022
Proof of Corollary
Denote the sets E, (IV(a)) V(+t)) 2 a()), 1 1= 1...., L, where
i()) F0/2v//1 1)/Oa(1). Note that lim liminf,r--e Pr(n R.) 2 1 Lo and
Pr(V(#t) 2 max V(+t) - 2n(1)) IsISL
=Pr(V(ft) - V(t)) + (((t)) 2 max V(+) - ((tt)) 20(1) + +
I
2 Pr((V(t)) - V(a))) ((+t)) 2
max V(tt)) V(At) -2((1) + -(()) E,) Pr())-1E))
1Stst.
PPr(V((t) a(1) 2
((tt) a(() 200))n== Ex) Pr(n
max isist
E,
where the last inequality holds because given the event n2 Ej, one has a(() S V(A)) - (AA))
and (mT) (ft) S (() for any l. This completes the proof by taking lim inf on both sides.
Proof of Theorem 2 To show the results in Theorem can be seen that
v VnT(O - 1)/0 (vitt)
V(TT))
DD (2) - ((
nT
8(1)
$
V 11001 -1
80t1 (7(
((+((1) (0) V(TT))
nf
0(0 1)
0 (1)(0))
VD... (5 10)) - (Vi (0)
nT
V 0(0 - 1)
80t1 (a)
(
+ B() O 1)
nT
Σ
Then by results in the proof of Theorem
I we can show that
lim Pr((I)= > 2a/2) = a.
(29)
nT 770000
This implies that
lim inf Pr (IV(=*) - -v(+t) 2a/2 2 V 1)a(1) B(I)
(30)
of +00
lim Pr((I) S 2n/22) = 11100 -a;
(31)
nf 400
which concludes our proof.
Proof of Corollary 2 We mainly show the proof of the second claim in the corollary, based on
which the first claim can be readily seen. Define an event such that < ( < L, (V - s and [V(=*) ) - ((+)) < za/(2L) (o/pli////TC/// - (1) + B(). Based on the 8-a. a. the following,
c(d) log( (i)/ /NT (i)/NT
assumption given in Corollary 2 and Theorem 2 we have lim P(E) 2 1-8 1-8-
-300
we suppose event E holds.
Inspired by the proofs of Corollary in (Mathe 2006) and Theorem 3 of (Su et al. 2020). we define i max(1= B()) < (t) + (1)). where ui(() za/(2L) 2a/(22) / 1)a(() Let uz(!) c(5) log( (I)(()))((())))))) By Assumption 7 for <i,
B(I) < B() S (1) (n),),
which further implies that for any 1 1
(#t) (( ) < B(I) + (1) S 21)
18

---

Under review as a conference paper at ICLR 2022
Then V(T*) E I() based on the construction of I()) for all / $ i. In addition, have for / S
[V - V(=*){ s 2u1 (1) + u2 (?),
(32)
by triangle inequality and event E. Since (1]) share at least one common element for S < i, we have i 2 Moreover, there must exist an element x such that x E [() n (()), where [v x -
ui (1) and [V(+t)) - - - (0). This indicates that
( - V( & (a,) x + (+t))) - x + (V(tt) v(T*))
(33)
S (1) 2u (7) S 3uz (7).
(34)
by again triangle inequality and Assumption 7 and
( - V(T*) - (2) + 3u, (1) & (2 + 3u, (1),
(35)
by event E and Assumption 7
Define |* = min(l: B()) + 11 (1) + u2(()) Then following the consider cases:
similar proof of (Su et al. 2020),
Case 1: If < i, then have
u2(() + B() - (1) s 21) + mz(1²) s 2m1 (1*) + 2B(1) uz()"),
where we use Assumption 7
Case 2: 1f > i, then we have
((u2(1) + (i) S (u2(1 + 1) + un (I+ s B( +1 s B(1),
where we use Assumption 7 This implies that
u un + B(i) S (1+11())(()
Combining two cases, show that
uz(() + (i) + B() S (1 (1+1/)(B(()) + +
+ u2((1));
< 1. Together with 33 we have
(V++)) - V(TT)| S ui) + 30)(1) 3 <3(1+1/(((B()) (1") u2())))
(36)
which concludes our proof.
C MORE DETAILS ON DQN ENVIRONMENTS
reward (+1)
(8)
(b)
(c)
(d)
Figure 5: DQN environments in our studies: (a) Er: FrozenLake-vO; (b) Ex: Banana Collectors (3D geomet- rical navigation task); (c) Eg: Pong-vO; ;;; (d) Eg: Breakout-vo v0; (c) Ex: Halfcheetah-vi; (f) Eg: Walker2d-vl.
We introduce our deployed DQN environments this section, which included four environments with discrete action (E, to Eạ) and two environments (Eg to Eg) with continuous action. These environments cover wide applications, including tabular learning (E,), navigation to a target object in geometrical space (Ez), digital gaming (Eg to E,). and continuous control (Eş to Eg).
Ej: Frozen Lake: The Frozen Lake a maze environment that manipulates an agent to walk from starting point (S) goal point without failing into the hole (H). We use FrozenLake-vd from OpenAl Gym (Brockman et al. 2016) as shown in Fig 5 a). We provide top-5 regret and precision results shown in Figure 6 and
19

---

Under review as a conference paper at ICLR 2022
Ragret Ro-55
Precision Tap-5
LAR
EC
0.
PPS EC
300 PEFOCC
0015
RPM
0020
INTEE
04
IPG-
Figure 6: Policy selection using top-k ranking re- gret score E, (Frozen Lake).
Figure 7: Policy selection using top-k ranking pre- cision in (Frozen Lake).
The Banana collector is one popular 3D 3D-graphical navigation environ- ment that compresses discrete actions and states
Ey: Banana Collector:
open source DQN benchmark from Unity ML.-Agents v0.3.(Juliani et al. 2018). The DRL. agent controls an automatic vehicle with 37 dimen- sions of state observations including velocity and ray-based perceptional information from objects around the agent. The targeted reward 12.0 points by accessing correct yellow bananas (+1) and avoiding purple bananas (-1) in first-person point of view as shown in Fig ((b). We provide the related top-5 regret and precision results shown Figure and
Protiata Rop-3
CON
Dov
0.10
PM64E I0Q
S1AC NS-WWC
PNG4RIC
0055
ITM
042
Numbers Taas
Numbers of Trjjs
Figure 8: Policy selection using top-k ranking re- gret score in Eg (Banana Collector).
Figure 9: Policy selection using top-k ranking pre- cision in Eg (Banana Collector).
075
PM54C
NS-C
0.2
INS-CC
0.0
KTQ
TEM
NGN
045
13H
Numbers Trajs
Nambers Trajs
Policy selection using top-k ranking re- gret score Es (Pong).
Figure 10:
Figure 11: Policy selection using top-k ranking pre- cision in Eg (Pong).
Ez: Pong: Pong is one Atari game environment from OpenAl Gym [Brockman et al. 2016 shown in ((). We provide its top-5 regret and precision results shown in Figure 10 and
as
2.023
DON MSS0O
0.5
J0v
0155
0.5
N6-0
KTQ
INSOCC SPAC
030
S3AC
NS-000
0.5
ADN
0853
330
200
400
25
200
200
400
Numbers Trass
Numbers of Trajs
Figure Policy selection using top-k ranking re- gret score in E, (Breakout).
Figure 13: Policy selection using top-k ranking pre-
cision in E, (HalfCheetah-vll
E.: Breakout: Breakout is one Atari game environment from OpenAl Gym (Brockman al. 2016) as shown Fig d). provide the related top-5 regret precision results shown in Figure 12
and 13
https://www.youtbbt
20

---

Under review as a conference paper at ICLR 2022
Top-3
Precismn 3
50N
075
MANOOV
20
0.5
040
BRAC
DMAC
PM 696
030
040
Numbers Trajs
Numbers of Trjjs
Figure 14: Policy selection using top-k ranking re- gret score in Es (HalfCheetah-vl).
Figure 15: Policy selection using top-k ranking pre- cision in Eş (HalfChectah--il
Top-
Precisan
DON
0.75
Iov
0.5
0.5
R10
604
035
BRAC
DMC
IEM
045
035
Numbers Trajs
Nambers of Trajs
Figure 16: Policy selection using top-k ranking re- gret score in Eg (Walker2d-vl).
Figure 17: Policy selection using top-k ranking pre- cision in Eg (Walker2d-vl).
Es: HalfCheetah-v1: Halfcheetah is a continuous action and state environment to control agent with monuments made by MuJoCo simulators as shown in Fig Sie). We provide the related top-5 regret and precision results shown in Figure 14d
Eg: Walker2d-v1: Walker2d-vl is continuous action and state environment to control agent with monuments made by MuJoCo simulators as shown in Fig f)).. We provide the related top-5 regret and precision results shown in Figure I6 and 17
C.I HTYPER-PARAMEN
INFORMATION
We select a total of 70 DQN based models for each environment. We will open source the model and implementation for future studies. Table I Table 2 and Table summarize their and setups. In addition, Figure and O selection in PMS experiments for the deployed DRL navigation task (Eg). From the experimental results, pessimistic (e.g., 0.001) is associated with slightly better attained top-5 regret. Meanwhile, the selection of O does not produce much different performance on selected policies but slightly affects the range of the selected policies.
and Figure 19 provide ablation studies on different scales of a and
Table 1: Hyper- parameters information for for DQN models used in E₁ Eg
Flyper-parameter
Values [I, 2}
Hidden layers
Hidden units
{16, 32, 64, 128) {1 x e-3 5 x o-1
Learning rate
DQN training iterations
(100, 500, 1k, 2k: (64)
Batch size
BROADER IMPACT
There also some limitations of the proposed PMS as of the preliminary attempts model selection for offline reinforcement learning. When the benchmarks environments (excluded Atari games) are based on simulated environments collect the true policy (Barth-Maron et al. 2018 Siegel et al.) 2019), more geal-world-bassee environments could be customized and studied in fu- ture works. For example, one experimental setup needs be carefully controlled in clinical set- tings (Tang & Wiens 2021) resilience-oriented (Yang et al. 2021) reinforcement learning.
21

---

Under review as a conference paper at ICLR 2022
Table Hyper-paramette information DQN models used in Eg E
Hyper-parametts
Values
Convolutional layers
2,3J
Convolutional units
116, 32}
Hidden layers
2,
Hidden units
{64, 256, 512)
Learning rate
(Ixe 5x e
DQN training iterations
(4M, 4.5,, 5M}
Batch size
644
Table Hyper-parameters information for double DQN (DDQN) models Van Hasselt al. 2016) with prioritized replay (Schaul et al. 2015) used Es to Eg.
Hyper-parameters
Values
Hidden layers
[4, 5, 6] {64, 128, 256, 512]
Hidden units
Learning rate
{1 11 x e-4 440M, 45M, 50M) (256)
DDQN training frames
Batch size
Buffer size
[00]
Updated target
(1000)
10"
012
010
105
030
034
132
0.0
6.0
20
40
Figure 18: Different for PMS selection.
Figure Different PMS selection.
22


Reviews:
[{"review_id": "uUlfA9tJDwQ", "reviewer": "Reviewer_MuG4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n- the paper tackles the important problem of model selection in ORL\n- the proposed method is based on religious guarantees, there are also two refined approaches for addressing the issue in an assumption\n- the simulation study shows the benefit of using the proposed approach\n\n**Weaknesses**\n- The claim “our approach essentially does not involve additional hyperparameter tuning” seems somewhat overselling. We should specify “O” and “\\alpha” to use PMS, and how they affect the model selection performance is unclear in the experiment\n\n- Some related papers are missing such as [a] and [b]\n\n[a] Mengjiao Yang, Bo Dai, Ofir Nachum, George Tucker, and Dale Schuurmans. Offline Policy Selection under Uncertainty. https://arxiv.org/abs/2012.06919 \n\n[b] Ilja Kuzborskij, Claire Vernade, András György, and Csaba Szepesvári. Confident Off-Policy Evaluation and Selection through Self-Normalized Importance Weighting. https://arxiv.org/abs/2006.10460 \n\n- there is room for improvement in the experimental section\n    - regret is defined in an additive manner, and Figure 3 contains results across different environments. Is this valid? Using relative regret ($\\frac{\\mathcal{V}\\left(\\pi_{l^{*}}\\right)-\\mathcal{V}\\left(\\pi_{l}\\right)}{\\mathcal{V}\\left(\\pi_{l^{*}}\\right)}$) seems more reasonable.\n    - What each point in Figure 4 means seems unclear\n    - I would love to see how the choice of “O” and “\\alpha” affect the model selection of PMS \n    - Figures 1 and 5 contain the result from only one of six environments. It is possible to use Appendix to share the results from the other environments that are not presented in the paper currently\n    - Additional evaluation metrics such as top-k presion used in [b] would make the experiments more practically relevant\n\nI would consider raising my score if I see substantial improvement in the experimental section. It is also essential to clarify the contributions of the paper compared to [a] and [b].\n", "summary_of_the_paper": "The paper studies the model selection problem in offline reinforcement learning (ORL). The model selection of ORL is challenging due to insufficient observational data from offline collection. The paper proposes a novel model selection approach to automate ORL development process and to identify a well-performed model given offline data. The proposed pessimistic model selection (PMS) method leverages an uncertainty quantification on value functions and a pessimistic idea. Some experiments show the superior performance of the proposed model selection method.", "summary_of_the_review": "The paper studies an important problem and provide nice solutions. However, some important related papers are missing and there are number of concerns in the experiment. Thus, I would recommend weak reject at this moment. I would like to see substantial effort, in particular the improvement in the experiments to change my score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "oKmy59lZ77g", "reviewer": "Reviewer_Sxiw", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This submission presented a pessimistic model selection approach (PMS) for offline deep reinforcement learning (ORL) and tested the approach with performance comparison on six simulated environments. Under several specified conditions, the authors provided asymptotic analysis to show that PMS can lead to the best model with respect the worst performance of the derived policy. \n\n**Strengths:** \n\n1. The authors proposed the PMS to select ORL model based on the constructed U(l) confidence interval estimate (page 6) by splitting offline training data and sequential estimation, which has the potential to address overfitting issues. Both theoretical analysis and empirical performance have been conducted. \n\n2. Two refined approaches were presented considering potential bias, even though empirically not showing advantages in simulated environments. \n\n**Weaknesses:** \n\n1. The authors may want to better explain some steps of the proofs in the appendix, for example, the last inequality for the proof of corollary 1 on page 17. \n\n2. Regarding empirical evaluation of PMS, it appears that the theoretical analyses were mostly based on several references, including Su et al (2020). In addition to comparing to OPE estimates in Tang & Wiens (2021), can the authors compare with other competing methods? Also, why the authors did not show the corresponding regrets for other competing methods in Figure 3 to better show the advantages of PMS? In Figure 3, what is $\\pi^*$? If it is the optimal policy, shouldn't the corresponding regret be zero? Also by simply looking at the tables in the appendix, it is not really clear how the initial models were selected in different experiments. More comprehensive performance evaluation should be provided. \n\n3. The submission may have to be checked carefully. For example, it is clear that the indices do not match in the last equation on page 7 for the constructed interval: should the index to be i instead of l in the righthand side?  In the beginning paragraph of Appendix A, the inequality directions are not consistent. The starting indices for t (from page 3) and o (from page 5) have to be checked to make them correct. \n\n4. The presentation needs to be improved. There are many language problems. For example, several acronyms were never defined, including OPE, which I assume that it stands for Offline Policy Evaluation. There are many grammar errors, for example, \"such A guarantee\" on page 2; \"such A procedure\" on page 8; \"...in continuous caseS\" on page 3. There are also incomplete sentences, for example, \"...where the refinement algorithms (PMS R1/R2) have only a relative +/- 0.423 % performance ...\" on page 9. Some referred indices were not correct. For example, in the paragraph before Assumption 3 on page 6, \"Assumption 1-->2 also requires that the convergence rates.... \" The language problems are almost everywhere in the appendix. For example, \"These environments covers wide applications includes tabular learning ....\", \"... for each environments.\" \"... one experimental setup needs careful control... \" The captions of Tables 2 and 3 (\"... prioritized replAy...\"). There are just way too many such problems all over the place. Finally, the authors stated on page \"Detailed results for each environment is given in Appendix B.\" But the appendix does not contain any results. \n\n", "summary_of_the_paper": "This submission presented a pessimistic model selection approach (PMS) for offline deep reinforcement learning (ORL) and tested the approach with performance comparison on six simulated environments. Under several specified conditions, the authors provided asymptotic analysis to show that PMS can lead to the best model with respect the worst performance of the derived policy. ", "summary_of_the_review": "The authors proposed the PMS to select ORL model based on the estimated worst performance to address potential overfitting issues in ORL. Both theoretical analysis and empirical performance have been conducted. The overall impression of the submission is that the authors should carefully check the writing and present more complete and comprehensive performance evaluation. Better presentation of the research results will be also appreciated. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "fcC4tus_eD6", "reviewer": "Reviewer_Y9ri", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "High-level comments:\n1. The theoretical part is weak, with many strong assumptions being made, but only providing asymptotic convergence results.\n2. The exposition can be significantly improved. The proposed method is based on many existing techniques like doubly-robust estimators, for which little explanation is provided. No self-contained pseudo-code is provided. Many notations are used without definition. \n\nDetailed comments:\n\n1. Please avoid using the acronym ORL for offline reinforcement learning, as online reinforcement learning can also be abbreviated with the same acronym.\n\n2. In related works, when discussing other works on model-selection in offline RL, please elaborate on their techniques and contrast with the current paper.\n\n3. $d^{\\pi,\\nu}$ is not defined.\n\n4. Could the authors elaborate on how equation (4) comes about? The RHS doesn't depend on the MDP transition at all, whereas the LHS clearly does, so how could they be equal?\n\n5. Section 4.1, I think you mean \"an estimate of $Q^{\\hat\\pi_\\ell}$ as $\\hat Q_\\ell$\", since there is no assumption that any of the $\\pi_\\ell$ is close to the optimal policy.\n\n6. Sec 4.2 in the description of the proposed algorithm, above equation (6), what do you mean by \"applying FQI on $D_0$ to compute $\\hat Q_\\ell$\"? To my knowledge, FQI only take as input a value function class $\\mathcal{F}$ but doesn't take any policy as additional input. If you mean to evaluate a particular policy $\\pi_\\ell$, the procedure is called Fitted-Q Evaluation (FQE). Also, it is confusing what \"candidate model\" means. Does it mean candidate policies returned by L different algorithms? The phrase \"model\" is very often confusing in RL, as many constructs are referred to as models, e.g. MDP transition in model-based RL or model as in a trained neural network. Avoid it if you can and be specific.\n\n\n7. $\\hat Q^{\\star (o)}_\\ell$ in (8) is not defined.\n\n8. What is the justification for constructing the confidence interval with finite samples based on an asymptotic convergence result? In finite sample regime, (11) can be arbitrarily far off.\n\n9. In what way can we see that large $O$ increases accuracy? Solely based on equation (11), the effect of large O on the convergence rate is negligible.\n\n10. $\\alpha$ is defined multiple times, in section 4 as the confidence parameter and in section 5 assumption 2 as the convergence rate.\n\n11. For section 6, could the authors elaborate on how their refined approach automatically adapts to unknown biases $B(\\ell)$? Also if there is bias in assumption 2 for policy improvement, then there definitely should be biases in assumption 3 for policy evaluation. Then, the additional assumption in Corollary 2 should not hold.\n\n12. No discussion about the computational complexity of the proposed algorithm. In particular, the min-max problem in doubly-robust estimation seems to scale poorly.", "summary_of_the_paper": "This paper proposed a new algorithm for model-section in offline RL based on adding the pessimism principle on top of existing OPE algorithms. Under strong assumptions, the paper provides an asymptotic convergence result for the proposed algorithm. It also provides empirical experiments demonstrating that the proposed algorithm outperforms existing baselines.", "summary_of_the_review": "The theoretical result is weak, and the exposition can be significantly improved.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "JCeRarEgy9U", "reviewer": "Reviewer_Wrcc", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Introduction mentions \"AutoML\" but I don't believe the main paper focuses on AutoML or even mentions it again. There seems to be a disconnect between this motivation and the content. \n\nSaying their proposed PMS approach is “tuning free” is kind of overselling: you still have to pick the  $O$ and $\\alpha$ hyperparameters, and additionally tuning the models architecture/training details for the value estimation procedure (probably FQI) that is used in each iteration to get $\\hat{Q}$. Consider adjusting this claim; also compare to Zhang & Jiang 2021 [1] (this paper is very new so the authors are not expected to know/cite it, it is however important to illustrate what tuning free might mean). \n\nMain results seem to have been aggregated over all 6 domains considered, however it is unclear whether the regret across domains are comparable i.e. on the same scale. It would be clearer to also provide per-domain results (the experiments section says these should be in Appendix B be but they are not). \n\nWhat are $O$ and $\\alpha$ set to for the experiments? \n\nSince the method splits the data into $O$ parts, this will lead to higher variance estimates for each iteration compared to using the full dataset. Is this accounted for in the theoretical analysis? \n\n[1] Siyuan Zhang, Nan Jiang. Towards Hyperparameter-free Policy Selection for Offline Reinforcement Learning. NeurIPS 2021. https://arxiv.org/abs/2110.14000", "summary_of_the_paper": "This paper considers the model selection problem in offline RL and proposes a new procedure called pessimistic model selection (PMS) that estimates performance lower bound for each candidate policy. Theoretical results show that under certain assumptions PMS can identify the best candidate policy with high probability. The proposed approach is compared to three common OPE methods used in Tang & Wiens 2021 and shows favorable performance on 6 RL domains. ", "summary_of_the_review": "This paper makes an attempt at an important problem - offline RL model selection, however some experimental details are missing and the results seem to be too brief and not as informative as they could be. It also has significantly smaller page margins than the ICLR required format. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a new approach called pessimistic model selection (PMS)  for model selection in offline RL and tests it in 6 different environments. Under certain assumptions this allows theoretical results that the best model is recovered with high probability.

Several points were raised by the reviewers and maintained after the rebuttal:
- Theoretical results were considered weak as they only hold asymptotically.
- Experimental results limited (potentially different regret scales, no sufficient comparison to other baselines).
- Exposition of the paper that needs to be improved.

Given the strong consensus among the reviewer I recommend rejecting this paper.
Decision: Reject