Model Selection for Off-policy Evaluation: New Algorithms and Experimental Protocol
Authors: Pai Liu
LingfengZhao
Shivangi Agarwal
Jinghan Liu
Audrey Huang
Philip Amortila
Nan Jiang
Venue: NeurIPS 2025 poster ()
Keywords: model selection
off-policy evaluation
Primary area: reinforcement_learning

Abstract: Holdout validation and hyperparameter tuning from data is a long-standing problem in offline reinforcement learning (RL). A standard framework is to use off-policy evaluation (OPE) methods to evaluate and select the policies, but OPE either incurs exponential variance (e.g., importance sampling) or has hyperparameters on their own (e.g., FQE and model-based). We focus on hyperparameter tuning for OPE itself, which is even more under-investigated. Concretely, we select among candidate value functions ("model-free") or dynamics models ("model-based") to best assess the performance of a target policy. We develop: (1) new model-free and model-based selectors with theoretical guarantees, and (2) a new experimental protocol for empirically evaluating them. Compared to the model-free protocol in prior works, our new protocol allows for more stable generation and better control of candidate value functions in an optimization-free manner, and evaluation of model-free and model-based methods alike. We exemplify the protocol on Gym-Hopper, and find that our new model-free selector, LSTD-Tournament, demonstrates promising empirical performance.

Model Selection for Off-policy Evaluation: New Algorithms and Experimental Protocol
Pai Liu
Lingfeng Zhao Columbia University
Shivangi Agarwal HITT Delhi
Jinghan Liu USTC
UIuC
Audrey Huang
Philip Amortila
Nan Jiang"
UIUC
UIUC
UIUC
Abstract
Holdout validation and hyperparameter tuning from data a long-standing problem in offline reinforcement learning (RL). A standard framework is to use off-policy evaluation (OPE) methods to evaluate and select between different policies, but OPE methods either incur exponential variance (e.g., importance sampling) have hyperparameters of their own (c.g., FQE and model-based). We focus on model selection for OPE itself, which is more under-investigater Concretely, we select among candidate value functions ("model-free") or dynamics models ("model-basede to best assess the performance of target policy. We develop: (1) new model-free and model-based selectors with theoretical guarantees, and (2) new experimental protocol for empirically evaluating them. Compared the model-free protocol in prior works, our new protocol allows for more stable generation and better control of candidate value functions in optimization- free manner, and evaluation of model-free and model-based methods alike. We exemplify the protocol on Gym-Hopper, and find that our new model-free selector, LSTD- ToUD-mourn,mnt,, demonstrates promising empirical performance.
Introduction
Offline reinforcement learning (RL) is promising paradigm for applying RL important application domains where perfect simulators are not available and we must learn from data 34 23 significant progress made in devising more performant training algorithms, how perform holdout validation and model selection-ap indispensable component of any practical machine learning pipeline-rent remains an open problem and has hindered the deployment of RL in real-life scenarios. Concretely, after multiple training algorithms (or instances of the same algorithm with different hyperparameter settings) have produced candidate policies, the primary task (which contrasts the secondary task which we focus on) is to select good policy from these candidates, much like how we select good classifierrregressor in supervised learning. do so, we may estimate the performance (i.e., expected return) of each policy, and select the one with the highest estimated return.
Despite the
Unfortunately, estimating the performance of different behavior policy is highly challenging task, known as off-policy evaluation (OPE). Popular OPE algorithms can be roughly divided into two categories, each with their own critical weaknesses: the first is importance sampling 49 22 (55). which has elegant unbiasedness guarantees but suffers variance that exponential the horizon, limiting applicability beyond short-horizon settings such as contextual bandits [36]. The second category includes algorithms such as Fitted-Q Evaluation (FQE) 14 31 47 marginalized importance sampling (38 44 57]] and model-based approaches
targer policy based data collected from
Conrespondence nanjiang@il inoii.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

[6]], which avoid the exponential variance; unfortunately. this comes at the cost of introducing their own hyperparameters (choice of neural architecture, learning rates, etc.). While prior works have reported the effectiveness of these methods [47], they also leave a chicken-and-egg problem: if these algorithms tune the hyperparameters of training, who tunes their hyperparameters?
In this work, we make progress on the latter problem, namely model selection for OPE algorithms themselves. We lay out and study a dichotomy of two settings: in the model-based setting, evaluation algorithms build dynamics models to evaluate target policy. Given the uncertainty of hyperparame- ters in model learning, we assume that multiple candidate models are given, and the task is to select one that best evaluates the performance of the target policy. In the model-free setting, evaluation algorithms only output value functions which we select from; see Figure (left) for visualization of the pipeline. Notably, these selection algorithms should be hyperparameter- freee themselves, to avoid further chicken-and-egg problems. Our contributions are:
New model-free selector (Section 3):
We propose new selection algorithm (or simply selector), for selecting between candidate value functions by approximately checking whether the function satisfies the Bellman equation. The key technical difficulty here is the
infamous double sampling problem M 53 [0], which typically requires additional function approximation (hence hyperparameters) to bypass. Our derivation builds on BVFT [62 64), which is the only existing selector that addresses double sampling in a theoretically rigorous manner without additional function approximation assumptions. Our new selector enjoys better statistical rates (n 1/2 vs. n -144) and empirically outperforms BVFT and other baselines.
2.
New model--asee selectors (Section 4) When comparing candidate models, popular losses in model-based RL (e.g.. Eg loss on next-state prediction) often exhibit biases under stochastic transitions (21). Instead, we propose novel estimators with theoretical guarantees, including novel adaptation of previous model-free selectors that require additional assumptions, where the assumptions are automatically satisfied using additional information in the model set (6 65].
3. New experiment protocol (Section 5): To empirically evaluate the selection algorithms, prior works often use FQE to prepare candidate Q-functions [64] (6). which suffer from unstable trainingPaand lack of control in the quality of the candidate functions. We propose a new experiment protocol, where the candidate value functions are induced from variations of the groundtruth environment: see Figure [1 (right) for an illustration. This bypasses the caveats of FQE and allows for the computation of Q-values in an optimization-free and controllable manner. Moreover, the protocol can also be used evaluate and compare estimators for the model-based setting. Implementation-wise. we use lazy evaluation and Monte-Carlo roll-outs to generate the needed Q-values. Combined with parallelization and caching, we reduce the computational cost and make the evaluation of new algorithms easier.
4. Preliminary experiments (Section o): We instantiate the protocol in Gym Hopper p and demonstrate the various ways in which we can evaluate and understand different selectors.
2 Preliminaries and Model Selection Problem
Markoy Decision Process (MDP). An MDP is specified by (S, (S,A,P, do). where S is the state space, .A is the action space, P: S x A- A(S) is the transition dynamics, R: S x A + [0, Rmax] is the reward function, 7 E [0, 1) is the discount factor, and do is the initial state distribution. A policy S - (AA) induces distribution over random trajectories, generated as so ~ dg. a; ~ T-(ss;). = R(st, at), S141 ~ P(-|st, az). We use [r and E-1 to denote such a distribution and the expectation thereof. The performance of a policy defined as J(T) = E, (E rrl, which is in the range of [0, Vmax] where Vina= := Rmaxx - 7).
Value Function and Bellman Operator. The Q-function Q" RSSA is the fixed point of T RSA -> RSSA, i.e., Qx = QTQ", where for any / E RSA (Tx [)(()) = R(s, (f(s) ,f(''TT)) We use the shorthand f(s', T) for Ea'= a')].
Ofr-policy Evaluation (OPE). OPE is about estimating the performance of a given target policy in the real environment denoted as M (S,A, P* RR, 7, do), namely Jaz- (T), using offline dataset D sampled from a behavior policy Ty. For simplicity, from now on we may drop the M
For example. our preliminary investigation has found that FQE often diverges with CQL-trained policies which is echoed by Nie et al. [46] in personal communications.
27
2

---

Pipeline in Practice (Model-free)
Experiment Protocol (Model-free)
M
M
Training Data
MS Data
Vary Env Parameters
Dats
Training Algo 2
Training Algo
MS Data
OPE
π
Mi
M2
M3
Data
MC Rollout (lazy eval)
FCE
F 2
FCE 3
Qi
Q2
JM- (
Q1
Q2
Q3
Ja (T)
OPE Error
Selector
OPE Error
Selector
Mr (TT
i
Esrulo (8,
Figure 1: An illustration of the pipeline in practice that motivates our research (left) and our proposed experimental protocol (right), both for the model-free setting: the pipeline for the model- based setting analogous and not visualized. Left: Training algorithms are run on training data to generate candidate policies, and choosing among them is the primary model-selection (MS) task (see Section which is not the focus of our work. These policies (e-g.. . become the target policies in OPE (e.g.. r), since accurate OPE can help solve the primary MS problem. Different FQE instances (e.g., with different hyperparameters, such as neural architectures) are used to approximate Q*, producing {Q The selector takes MS data and {Qi} } as input and choose one of them to estimate ((T) JM- (TT).
Right: Illustration our protocol for an experiment unit (Section 5). The target policies a can be learned from separate training data but can also be produced in other ways, such training on inaccurate models M,, which can realistic for practical scenarios with inaccurate simulators. {M.) is prepared by varying environment parameters. Monte-Carte Carlo rollouts are used to generate the Q-values for the data points in the MS data, which avoids potentially unstable optimization in the OPE pipeline; this is the source of stability and controllability compared to the prior protocol that mimics the practical pipeline. For further discussion on the limitation of our protocol and the trade-offs, Section
in the subscript when referring to properties of M e.g., = (x) Q _ standard simplification, our theoretical derivation assumes that the dataset consists of i.idd tuples (s, r') generated as (s, a) μ. R(s a), s' (* a). We use Ex,[1] to denote the true expectation under the data distribution, and ED[1] denotes the empirical approximation from D.
Q7 etc. As
Model Selection. We assume that there are multiple OPE instances (e.g. OPE algorithms with different hyperparameters) that estimate J(T), and goal is to choose among them based on the offline dataset D; see Figure [ for visualization. Our setup is agnostic w.r.t. the details of the OPE instances, and views them only through the Q-functions the dynamics models they produce Concretely, two settings are considered:
Model-free: Each OPE instance outputs Q-function. The validation task is to select Q from the candidate Q-functions Q {Q4) (0e)e((m) [m]] 4 such that E, [Q(s, T)] [TT). We are interested in the regime where at least one of the OPE instances produces a reasonable approximation of Q i.e., 3i, Q2 Q while the quality of other candidates be arbitrarily poor. We make
3 For alternative formulations such as selecting function class OPE algorithm, see Appendix 'In practical scenarios, (Qe)emm) and [M, )iem)) may be learned from data, and we assume 2 is a holdout dataset independent of the data used for producing {Mi Jeemm) and (Q² Jse(m].

---

the simplification assumption Q E Q for theoretical derivations, and handling misspecification (QT & 2) is routine in recent RL theory and omitted for presentation purposes B We will also test our algorithms empirically in the misspecified case (Section 6.2
Model-based: Each OPE instance outputs an MDP M; and uses JM, (T)) as an estimate of
[z).. W.I.o.g. assume M, only differs from M* in the transition P., handling different
reward functions straightforward. The task is to select M from M := (M.)elm) such that (x) (T)) Similar the model-free case, we assume M E M in the derivations.
The model-free algorithms have wider applicability, especially when we lack structural knowledge of the dynamics. A model-free algorithm can always be applied to the model-based setting: any {MI...., Mm } induces Q-function class {Q7 (Q,,..... ...,.... Q3 which can be fed into model-free algorithm (as demonstrate in our protocol; Section and M M implies Q -QM- Q.
3 New Model-Free Selector
In this section we introduce our new model-free selector, L.STD-Tourament, To start, we review the difficulties in model-free selection and the idea behind BVFT 62 64 which we build on.
3.1 Challenges of Model-free Selection and BVFT
To select Q from Q = (Q...... Qm perhaps the most natural idea is to check how much each candidate function Qi violates the Bellman equation Q" T==Q, and choose the function that minimizes such a violation. This motivates the Bellman error (or residual) objective:
Eu[(Q: Q=).
Unfortunately, this loss cannot be estimated due to the infamous double double-sampling problem 10 and the naïve estimation, which squares the TD error, is a biased estimation of the Bellman error (Eq. M in stochastic environments:
(TD-sq) E (s) - r r( +))
Common approaches to debiasing this objective involve additional "helper" classes (e.g., in Section[4, making them not typerparameter-free. See also Appendix [A] a detailed discussion of how existing works tackle this difficulty under different formulations and often stronger assumptions.
The idea behind BVFT 62 is to find an OPE algorithm for learning Q" from a function class F, such that achieve polynomial sample-complexity guarantees, it suffices if F satisfies 2 assumptions: (1) Realizability, that Q" E F. (2) Some structural (as opposed to expressivity) assumption on F, e.g., smoothness, linearity, etc. Standard learning results in RL typically require stronger expressivity assumption than realizability, such as the widely adopted Bellman-compe Bellman-completeness assumption (T^ E F. F). However, exceptions exist, and BVFT shows that they can be converted into a seirwis--comprisso subroutine for selecting between two candidates {Qi, 2ih and extension to multiple candidates can be done via tournament procedure. Crucially, we can use {Q., } to automatically create an F needed by the algorithm without additional side information or prior knowledge. We will demonstrate such process in the next subsection.
BVFT.
In short, BVFT provides a general recipe for converting a special kind of "base" OPE methods into selectors with favorable guarantees. Intuitively, the "base" method/analysis will determine the properties of the resulting selector. The "base" of BVFT is -r-irrelevant abstractions (35] 20). where the structural assumption on F is being piecewise-constane Our novel insight is that for learning Q*, there exists another algorithm, LSTDQ (29), which satisfies the needed criteria and has superior properties compared to Q -irreleevnnt abstractions, thus can induce better selectors than BVFT.
3.2 LSTD-Tournament
We now provide a theoretical analysis of LSTDQ (which is simplified from the literature and show how to transform it into selector via the BVFT recipe. In LSTDQ, linear function approximation, i.e., it is assumed that a feature map Φ : S A + Rd is given, such that Q* a) o(s.a) 0*, where d Rd is the groundtruth linear coefficient. Equivalently, this asserts that the induced linear class, Fo == {Ø(-) T Id satisfies realizability, Q" F
42 48)
learn Q" via

---

LSTDQ provides a closed-for estimation of 0* by first estimating the following moment matrices:
Σ = (,, ())((s)((())))) a) ] Ex : En( m) }) A = E >2"
b = a))
(4)
As simple algebraic fact, A0 b. Therefore, when A is invertible, immediately have that 8" b. The LSTDQ algorithm thus simply estimates and b from data, denoted A and respectively, and estimate as '.. Alternatiely,, for any candidate 8, || - bllcc can serve as a loss function that measures the violation of the equation A = 6, which we minimize over.Iss simite-sample guarantee given below. All proof's of the paper can be found in Appendix C
Theorem 1. Let C IR be a set of parameters such that 6* E O. Assume max (alssaalll((s.. ( a)ll S B and maxaçe | < 1. Let 0 arg mingee ||A0 blloo. Then, with probability at least - 8,
l|Q - -T z mmxx Ba
d
(5)
Tmin (A)
n
where m-in(.) (.) is the smallest singular value.
Besides the realizability of Q", the guarantee also depends on the invertibility of A, which can be
viewed coverage condition, since A changes with the data distribution l 2 3
23 In fact, in the
on-policy setting, minn (A) can be shown to be lower-bounded away from 42 see Appendix C.5
LSTD- TSUD-Tournament. We are now ready describe with the case of two candidate functions, {Qa, 22 ). where Q E [Qi, 22 To apply the LSTDQ algorithm and guarantee, all need is to create the feature map & such that Q is linearly realizable in ø. In the spirit BVFT, we design the feature map as
selector. Recall that first deal
$1g(s, a) ())
(6)
The subscript " makes clear that the feature is created based on Qi and Qj as candidates, and we will use similar conventions for all quantities induced from di.j. e.g., Aiji bi.j. etc. Obviously, Q" is linear in di,j with H* E {[11 0]_ ]. Therefore, to choose between Q; and Qj, we can and choose the one with smaller calculate the LSTDQ loss of [1, 0] and [0, 1] under feature di,5
loss. For 0 = [], we have A, bisj
(,(s,a) (5)(s (1Qr(s, o), Q2 a)] 10, ()..s,,),11)) 2,() m))}} !!
E.
(7)
(s. a)
- (x), r] ! E, 00
(8)
Taking the infinity- of the loss vector, we have
H
|A
buolloo max kE(i.)
-r -
(9)
The loss for 0 = [0, 1]T is similar, where Q; is replaced by Q₁. Following BVFT, we can generalize the procedure m candidate functions (Q1....,Q... by pairwise comparison and recording the worst-case loss, this leads to our final loss function:
L(Q (Q) T) max (a.(v(sss)) - (r)((((',m)))
(10)
kE [ms]
The actual algorithm replaces Ey, with the empirical estimation from data, and chooses the Q, that minimizes the loss. Building on Theorem we have the following guarantee:
Theorem 2. Given Q" = Q E (Qi)iemm) the Q7 that minimizes the empirical estimation of L(Q;; (Q, [j((m] T) (Eq. (1O)) satisfies that w.p. 2 1 1 8,
max 24V3 max
log(8m/o)
(J(T) Fondd (0,(8,
(11)
iE[m]\(P) m (A,e)
72
5

---

Comparison to BVFT [62]. BVFT's guarantee has a slow n 1/4 rate for OPE [64] 19,, whereas our method enjoys the standard n 1-12 rate. The difference is due to an adaptive discretization step in BVFT, which also makes its implementation somewhat complicated as the resolution needs to be heuristically chosen. By comparison, the implementation of LSTD-Tournamen is simple and straightforward. Both methods inherit the coverage assumptions from their base algorithms and are not directly comparable, and a nuanced discussion on this issue can be found in Appendix C.5
Variants. A key step in the derivation is to design the linearly realizable feature of E but the design is not unique as any non-degenerate linear transformation would also suffice. For example, can use Φıح (Q+/c,, (Q; Qi) 2ej.c. the "diff-of-value" term Qj - Qi has shown improved numerical properties in practice [27 12), and Gi, CJ.i can normalize the discriminators to unit variance for further numerical stability; this will also be the version we use in the main-text experiments.
Empirical comparison across these variants can be found in Appendix E.2
4 New Model-Based Selectors
We now consider the model-based setting, i.e., choosing a model from {M; Jefmm such that Jgy(T) (T) R (=)) This can be practically relevant when we have structural knowledge of the system dynamics and can build reasonable simulators, but simulators of complex real-world systems will likely have many design choices and knobs that cannot be set from prior knowledge alone. As alluded to the end of Section model-free methods can always be applied to the model-basee setting by letting
Q = {Q7,, hie/m. The question is: are there methods that leverage the additional side information in .M (that is not in Q) to outperform model-free methods?
We study this question by developing algorithms with provable guarantees that can only be run with M but not Q (which means they must be using additional information), and include them in the empirical comparisons in Section 6 To our surprise, however, these algorithms are outperformed by LSTD- Tournament, which is simpler, computationally more efficient, and more widely applicable. Despite this, our model-based development produces novel theoretical results and can be of independent interest, and also provides additional baselines for LSTD Tournamett in empirical comparison. We briefly describe the studied methods below, with details deferred to Appendix B
Naïve Baseline. A natural method is to use the model- loss: any model M is scored by (ssaas")>>p >-p.s.pppppppp-a I||s' =]]], where P is the dynamics of M, and || is some norm (e.g.. lg norm) that measures the distance between states. Despite its wide use in the model-based RL literature, the loss has a crucial caveat that it exhibits bias towards more deterministic systems, which problematic when the groundtruth environment is stochastic. We will see empirical evidence of this in Section 6 (see "ib_naivee" in Figures and 4.
2. Regression- Selector (Appendix B.1 Recall from Section 3 that main difficulty in directly estimating the Bellman error the lack of access to T Antos et al. 6 suggest that * f for any f can be learned via regression f E arg min geg FT, (g)(s, a) >f ?), provided that T f E G. Zitovsky et al. [65] apply this idea to the model selection problem and uses user-provided G, which requires additional hyperparameters
model-free selector. However, in Appendix [.I we show that the G class can be constructed automatically from M, making it a strong baseline (due its of additional information in M) for our L.STD-Tournameet in the empirical comparison ("mb Zitovsky et al." and "mb Antos et al" in Figure 4
3. Sign-flip Selector (Appendix B.2). We also develop a novel selector that takes the spirit of prreessionbbaedd selector, but replace its squared loss with absolute value. It comes with new theoretical guarantee (Theorem 4), and is implemented in the experiments Limb_sign___ _in"_
5 A Model-Based Experiment Protocol
Given the new selectors, we would like to evaluate and compare them empirically. However, as alluded to in the introduction, current experiment protocols have various caveats and make it difficult to evaluate the estimators in well-controlled settings. In this section, we describe a novel model-based experiment protocol, which can be used to evaluate both model-basee and model-free selectors.

---

5.1 The Protocol
Our protocol consists of experiment textitunits defined by the following elements (see Figure TR):
1. Groundtruth model M
2. Candidate model list M {M, Nejm)-]-
3. Behavior policy my and offline sample size n.
4. Target policies II {mI,...,T)}>
Given the specification of unit, we will draw dataset of size from M using behavior policy m For each target policy т E II, we apply different selectors to choose a model M E .M to evaluate т. Model-free algorithms will access M only through its Q-function, Q7+ effectively choosing from the set = (Qhs M E M). Finally, the prediction error [JM( JM- (T)) ] is recorded and averaged over the target policies in II. Moreover, we may gather results from multiple units that share the same M but differ M and/or the behavior policy to investigate issues such as robustness to misspecification and data coverage,
will demonstrate in the next section.
Lazy Evaluation of Q-values via Monte Carlo. While the pipeline is conceptually straightforward. practically accessing the Q-function Qh is nontrivial: we could run TD-style algorithms in M to learn Q% but that invokes a separate RL algorithm that may require additional tuning and verification, and it can be difficult to control the quality of the learned function.
Our innovation here is note that, for all the model-free algorithms we are interested in evaluating. they all access Q%, exclusively through the value of QM a) and (s' TT) for (s, s') in the offline dataset D. That is, given n data points D, we only need know 2n scalar values about Q7,- Therefore, we propose to directly compute these values without explicitly representing Q7r and each value can be easily estimated by averaging over multiple Monte- Carlo rollouts [50], i.e. QT (8.a) a) = Ex 1> 7 = 8, do a) can be approximated by rolling out multiple trajectories starting from (s, a) and taking actions according to æ. For the model-based estimators proposed in Section we need access to quantities in the form of (TM, Qhs,) (s, a) (see Appendix B). This value
can also be obtained by Monte-Carlo simulation: (I) start in (s, a) and simulate one step in Mj. then
(2) switch to M,, simulate from step 2 onwards and rollout the rest of the trajectory.
5.2 Computational Efficiency
Despite not involving neural-net optimization, the experiment can still be computationally intensive due to rolling out a large number trajectories. In our code, we incorporate the following measures to reduce the computational cost:
Q-caching. The most intensive computation is roll-out Monte-Carlo trajectories for Q-value estimation; the cost of running the actual selection algorithms is often much lower and negligible. Hence, we generate these MC Q-estimates and save them to files, and retrieve them during selection. This makes it efficient to experiment with new selection algorithms add extra baselines, and also enables fast experiment that involves subset of the candidate models (see Section 6..2)
Bootstrapping. To account for the randomness of 'D, we use bootstrapping to sample (with replacement) multiple datasets, and report an algorithm's 's mean performance across these bootstrapped samples with 95% confidence intervals. Using bootstrapping maximally reuses the cached Q-values and avoids the high computational costs of sampling multiple datasets and performing Q-caching in each of them, which is unavoidable if we were to repeat each experiment verbatim multiple times.
Exemplification of the Protocol
In this section instantiate our protocol in the Gym Hopper environment to demonstrate its utility, while also providing preliminary empirical results for our algorithms. Our code is available at https://pp www.ttttu.....eeeeeeeeeeeeeeeeee
6.1 Experiment Setup and Main Results
Our experiments will be based on the Hopper-v4 environment 99. To create a variety of environments, add different levels of stochastic noise in the transitions and change the gravity constant (see
7

---

MFG
Monte-Carlo Estimates
Figure 2: Left: ( in
M E Mg (cf. Section
140
150
Return
6.1] for different target poli- cics. Right: Convergence
120
100
100
of Monte-Carlo estimates of ((T). Each curve corresponds to target policy.
80
40
20
200 Number of Rollouts
100
Gravity
Appendix[D.I] Each environment then parameterized by the gravity constant g and noise level n. We consider arrays of such environments as the set of candidate simulator M: in most of our results,
we consider a "gravity grid" (denoted using MFG in the figures) Mg == {MS g ..... Mg (fixed noise level, varying gravity constants from -51 to -9) and a "noise grid" (MEN) Mn = {MM..... [M.) M11)
(fixed gravity constant, varying noise level from 10 100). Each array contains 15 environments, though some subsequent results may only involve a subset of them (Section 6.2). Some of these simulators will also be treated as groundtruth environment M which determines the groundtruth performance of target policies and produces the offline dataset D.
Behavior and Target Policies. We create 15 target policies by running DDPG [37 in one of the environments and take checkpoints. For each M the behavior policy is the randomized version of one of the target policies; see Appendix D.2 for details. A dataset is collected by sampling trajectories until n = 3200 transition tuples are obtained. As a sanity check, we plot (M(x) (T for и 1lg and M E Mg in Figure 2 As can be shown in the figure, the target policies have different performances, and also vary in a nontrivial manner w.r.t. the gravity constant g. It is important to perform such a sanity check to avoid degenerate settings, such as Ja (T) varies little across M M (then even random selection will be accurate) or across т E II.
D..2
Number of Rollouts We then decide the two important parameters for estimating the Q-value, the number of Monte-Carlo rollouts and the horizon (i.e., trajectory length) H. For horizon, we set H 1024 which is substantially longer than typically observed trajectories from the target policies. For we plot the convergence of Jar (T) estimation and choose = 128 accordingly (see Figure 2)).
Compared Methods. We compare our methods with baselines, including TD-square (Eq naive model-based (2q. BVFT 64), and "average Bellman error" (Er(Qi(s,a)
7 Qi(ss, (((s'mm) T]]
25 which can be viewed as our bodel-based methods in Section 4 require MC rollouts for (TM, Q5, : i.j [m]), which requires ((m²) computational complexity. Therefore, we first compare other selectors (mostly model-free) in Figure 3 with m 15; the relatively large number of candidate simulators will also enable the
but with trivial constant discriminator. The
MF.:: 0=100.0
MFG.
MF.G: gm-8, o=100.0
12.5
mf Logg bellmer eeror
10.0
10-
30-
mL, Jston, hoummuemmmttttt buft
5.0
20
15.
Figure 3: Main results for comparing model-free selec- tors in the gravity grid (MEG; top row) and the noise grid (MEN: bottom row). Each plot corresponds to a different M as indicated in the plot title. "mb naive" is model- based but still included since it does not require Bellman op- erator rollouts.
0.0
OPE Bror
1000
Sample Size
2000
3000
1000
Sample Size
2000
3000
0
1000
Sample Size
2000
3000
MFN: gm-30, p=10.0
MFN gm-30, 5555.0
MEN: o-) 10.00
5-
2-
3-
2-
10000
3000
2000
Sample
Sample
Sample
8

---

MB.S: 9--36, 0-100.0
MB.6 g--30, 0=100.0
MB.G: 9=:24, e=100.0
random
3.00
mf bids tournament
nalve
2.50
sgn
mb Anteetos ct
2,00
Figure 4: Main results for comparing model-based selec- tors (ef. Appendix tails of methods). LSTD- Tournament is included the best model-free selector for comparison, which surpris- ingly outperforms the more so- phisticated model-basee ones in Section
OPE Ener
3000
1000
2000 Sample
Sample
gm-30,
9m 0m55.0
gm-30,
B for de-
10000
2000
3000
1000
2000
3000
0
1000
2000
3000
Sample Size
Sample Size
Sample Siz0
g--30, 1010.0
gm-51,
2.0
randdom square
OPE Error
OPE Error
155
mf. avg. www.bbblllaceeec error
mfdit toumament
10-
mf. bvft
20
40
5
10
0.0
0.5
1.0
Noise Gap
Misspectfication Level
Ratio of Targ Data
Figure 5: Left: OPE error vs. simulator gaps. Middle: OPE error vs. misspecificcation Right: OPE етrо vs. data coverage.
later subgrid studies in Section 6.2 We then perform a separate experiment with m 5 for the model-based selectors (Figure44
Main Results. Figure 3 shows the main model-free results. Our LSTD- Toummament method demonstrates strong and reliable performance. Note that while some methods sometimes outperform it, they suffer catastrophic performances when the true environment changes. For example, the naïve model-based method performs poorly in high-noise environment (a 55.0 and 100.0) when the candidate models have varying degrees of stochasticity (MEN), as predicted by theory (Section 4 BVFT's performance mostly coincides with TD-sq, which possible degeneration predicted by (64). This is particularly plausible when the number of data points n is not large enough to allow for meaningful discretization and partition of the state space required by the method.
see Appendix D.2 where implement the three model-based selectors in Section whose computational complexities grow quadratically with [M Our expectation was that (1) these algorithms should address the double- sampling issue and will outperform na've model-based when the latter fails catastrophically, and (2) by having access to more information, model-based should outperform model-free algorithms under realizability. While the first prediction is largely verified, prediction went wrong, and our LSTD-Towr Tournament method is more robust and generally outperforms the more complicated model-based selectors.
Figure shows the result on smaller candidate sets (MB.G and MB.N;
surprised find that the second
6.2 Subgrid Studies: Gaps and Misspecifications
We now demonstrate how to extract additional insights from the Q-values cached earlier. Due to space limit
only able to show representative results in Figure and more comprehensive results can found in Appendix(Ei
Gaps. We investigate an intellectually interesting question: is the selection problem easier if the candidate simulators are very similar to each other, or when they are very different? We argue that the answer is neither, and intermediate difference (or gap) is the most challenging: if the

---

simulators are too similar, their JM (T) predictions will all be close to Jas- (T) since M = M and any selection algorithm will perform well; the simulators are too dissimilar, it should be easy to tell them apart, which also makes the task easy.
To empirically test this, we let M M, and run the experiments with different 3-subsets of Mn. including {6, 7, 8) (least gap). {5, 7,9). 7,9).... {0, 7, 14} (largest gap). Since the needed Q-values have already been cached in the main experiments, we can skip caching and directly run the selection algorithms. We plot the prediction error as function of gap size in Figure SL, and observe the down-U curves (except for trivial methods such as random) as expected from earlier intuition.
Miissecciiintiinn Similarly, we study the effect of misspecification, i.e., M & .M. For example, we can take M .M.. and consider different subsets of Mn 0-4 (realizable), 1-5 (low misspecifi- cation).... 10-14 (high misspecification) Figure F plots OPE error vs. misspecification level, where expect to observe potential difference in the sensitivity to misspecification. The actual result is not that interesting given similar increasing trends for all methods.
6.3 Data Coverage
In the previous subsection, we have seen how multiple experiment units that only differ in M can provide useful insights. Here we show that we can also probe the methods' sensitivity to data coverage by looking at experiment units that only differ in the dataset D. In Figure (5,, we take previous experiment setting (Mg) and isolate a particular target policy и,; then, we create two datasets: (1) Dj sampled using æ: (2) Dorr sampled using a policy that is created to be very different from the target policies and offer very little coverage (see Appendix D.2p. Then, we run the algorithm with A fraction of data from D, combined with (1 A) from Dott' as predicted by theory, most methods perform better with better coverage (large X), and performance degrades as À goes to O.
7 Limitations, Future Directions, and Conclusion
We conclude the paper with a discussion of the limitations of our work and potential future directions. Realism of Candidate Q-functions.
While our proposed experimental protocol offers significant advantages in controllability and stability (c.f. Footnote [ a key limitation lies in the realism of the candidate Q-functions it generates. In practice, these functions will likely come from learning algorithms ge. TD/FEE,, whose errors may differ from the structured errors induced by our protocol (e.g., varying gravity uniformly across states). Varying environment parameters in more complex, state-dependent ways could potentially generate more realistic error patterns, presenting an interesting avenue for future investigation. That said, the prior protocol using FQE may also face realism challenges, just in a different way: in practice, we carefully tune the optimization of OPE algorithms for each target policy to produce reasonable candidates, which can be infeasible in empirical benchmarking given the sheer number of policies we are working with, resulting in poorer candidates. Neither approach perfectly mirrors reality. In addition, our protocol offers controllable quality that enables targeted studies (like the gap experiments in Section (6.2). so for more comprehensive empirical studies, comparing results from both protocols is advisable.
Fundamental Theory of LSTDQ. As discussed Appendix C.5 our theoretical analysis of LSTD- Tournament reveals limitations and open questions for the coverage assumption made by LSTDQ (amin(A) in Theorem D). which is standard in the literature and inherited by LSTD-Toumament. Such conditions differ significantly from the more standard concentrability coefficients (c.g., C% in Theorems 3 and and the lack of satisfactory understanding of such a classic algorithm calls for future investigation. We hope study this question in the future, and any progress would directly improve the guarantees and understanding of SSTDD-Touramme..
As another potential future direction, LSTDQ can be viewed as an application of instrumental- variable regression in value-function estimation [ where the left ø in the moments of (3. plays the role of an instrument, While such choice is standard and natural, it will be interesting to explore alternative choices of instruments and examine if they can lead to improved theoretical guarantees and practical performance in the model-selection problem.
10

---

Acknowledgements
Nan Jiang acknowledges funding support from NSF CNS-21 124711, NSF CAREER IIS-2141781, Google Scholar Award, and Sloan Fellowship.
References
[1] Alckh Agarwal, Sham Kakade, Akshay Krishnamurthy, and Wen Sun. Flambe: Structural complexity and representation learning of low rank mdps. arXiv preprint arXiv:2006.108// 10814, 2020.
[2] Philip Amortila, Nan Jiang, and Tengyang Xie. A variant of the wang-foster-kakan lower bound for the discounted setting. arXiv preprint 2.107520 2020.
[3] Philip Amortila, Nan Jiang, and Csaba Szepesvári. The optimal approximation factors in misspecified off policy value function estimation. In International Conference on Machine Learning, pages 768-790. PMLR, 2023.
[4] Philip Amortila, Tongyi Cao, and Akshay Krishnamurthy. Mitigating covariate shift in misspec- ified regression with applications to reinforcement learning. arXiv preprint 2xXi:2441.112216 2024.
[5] Philip Amortila, Dylan J Foster, Nan Jiang, Akshay Krishnamurthy, and Zakaria Mhammedi. Reinforcement learning under latent dynamics: Toward statistical and algorithmic modularity. arXiv preprint arXiv:2410.1799 2024.
[6] András Antos, Csaba Szepesvári, and Rémi Munos. Learning near-optimal policies with Bellman-residual minimization based fitted policy iteration and a single sample path. Machine Learning, 2008
[7] Leemon Baird. Residual algorithms: Reinforcement learning with function approximation. In Machine Learning Proceedings 1995, pages 30-37. Elsevier, 1995.
[8] Steven J Bradtke and Andrew G Barto. Linear least-squares algorithms for temporal difference learning. Machine learning, 22(1):33-555 57, 1996.
[9] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jic Tang, and Wojciech Zaremba. OpenAl gym. arXiv preprint 1rXiv:1606.01540 2016.
[10] Jinglin Chen and Nan Jiang. inormatiio--hheorede considerations in batch reinforcement learning. In Proceedings of the 36th International Conference on Machine Learning, pages 1042-1051, 2019.
[11] Yutian Chen, Liyuan Xu, Caglar Gulcehre, Tom Le Paine, Arthur Gretton, Nando De Freitas, and Arnaud Doucet. On instrumental variable regression for deep offline policy evaluation. Journal of Machine Learning Research, 23(302): 1040, 2022.
[12] Ching-An Cheng, Tengyang Xie, Nan Jiang, and Alekh Agarwal. Adversarially trained actor critic for offline reinforcement learning. International Conference on Machine Learning, 2022.
[13] Google Deepmind. Mujoco documentation. URL
www.:ecoo.....rccc...cc.c....... ///en/
stable/ wwwpootttttooinnooooooooooooooee wttp/iiidxxxiieem html
[14] Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6:503-556, 2005.
[15] Amir-massoud Farahmand and Csaba Szepesvári. Model selection in reinforcement learning. Machine learning, 85(3):299-332, 2011.
[16] Scott Fujimoto, David Meger, Doina Precup, Ofrr Nachum, and Shixiang Shane Gu. Why should trust you, bellman? the bellman error is a poor replacement for value error. arXiv preprint arXiv:2201. 12411 2022.
[17] David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv:1803.10122.1 2018.

---

[18] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.0411 2023.
[19] Zeyu Jia, Alexander Rakhlin, Ayush Sekhari, and Chen-Ye Yu Wei. Offline reinforcement learning: Role of state aggregation and trajectory data. arXiv preprint arXiv:2403. /7091, 2024.
[20] Nan Jiang. CS 598: Notes on State Abstractions. University of Illinois at Trbana-Champpinn 2018. http:// http///aajiaaag cs www.:illiii.s...m.m https//sseeeeielleeeeeee/////eee pdr
[21] Nan Jiang. note on loss functions and error compounding in model-based reinforcement learning. arXiv preprint M.Xiv:24044009446 2024.
[22] Nan Jiang and Lihong Li. Doubly Robust Off-policy Value Evaluation for Reinforcement Learning. In Proceedings of the 33rd International Conference on Machine Learning, volume 48, pages 652-661, 2016.
[23] Nan Jiang and Tengyang Xie. Offline reinforcement learning in large state spaces: Algorithms and guarantees. 2024. https://nanjiang cs. wllinoisi....... /ttt/eeeeeee//SSSSeeeeeeeee Sasue_Offline RL. pdf
[24] Nan Jiang, Alex Kulesza, and Satinder Singh. Abstraction Selection in Model-based Reinforce- ment Learning. In Proceedings of the 32nd International Conference on Machine Learning, pages 179-188, 2015.
[25] Nan Jiang, Akshay Krishnamurthy, Alekh Agarwal, John Langford, and Robert E Schapire. Contextual decision processes with low Bellman rank are PAC-learnable. In International Conference on Machine Learning, 2017.
[26] Haruka Kiyohara, Ren Kishimoto, Kosuke Kawakami, Ken Kobayashi, Kazuhide Nakata, and Yuta Saito. Scope-rl: A python library for offline reinforcement learning and off-policy evaluation. arXiv preprint arXiv:2311. 2023.
[27] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33: 1179-1191, 2020.
[28] Aviral Kumar, Anikait Singh, Stephen Tian, Chelsea Finn, and Sergey Levine. workflow for offline model-free robotic reinforcement learning, arXiv preprint arXiv:2109.10813 2021.
[29] Michail G Lagoudakis and Ronald Parr. Least-squares policy iteration. The Journal of Machine Learning Research, 4:1 107-1149, 2003.
[30] Alessandro Lazaric, Mohammad Ghavamzadeh, and Rémi Munos. Finite-sample analysis of least-squares policy iteration. The Journal of Machine Learning Research, 13(1):304133774 2012.
[31] Hoang Le, Cameron Voloshin, and Yisong Yue. Batch policy learning under constraints. In International Conference Machine Learning, pages 3703-3712, 2019.
[32] Jonathan Lee, George Tucker, Ofir Nachum, and Bo Dai. Model selection in batch policy optimization. In International Conference on Machine Learning, pages 12542-12569. PMLR, 2022.
[33] Jonathan N Lee, George Tucker, Ofir Nachum, Bo Dai, and Emma Brunskill. Oracle inequal- ities for model selection in offline reinforcement learning. Advances in Neural Information Processing Systems,
2022.
[34] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint 2xXiv:200500164 2020.
[35] Lihong Li, Thomas J Walsh, and Michael I. Littman. Towards unified theory of state abstraction for MDPs. In Proceedings of the 9th International Symposium on Artificial Intelligence and Mathematics, pages 531-539, 2006.
12

---

[36] Lihong Li, Wei Chu, John Langford, and Xuanhui Wang. Unbiased Offline Evaluation of bandextual-badit-ta News Article Recommendation Algorithms. In Proceedings of the 4th International Conference on Web Search and Data Mining, pages 297-306, 2011.
[37] Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Manfred Otto Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforce- ment learning. CoRR, abs/
semanti 2015. URL.ttts: semanticscholaar cscholar org/
CorpusID: 16:32667666666 16326763
[38] Qiang Liu, Lihong Li, Ziyang Tang, and Dengyong Zhou. Breaking the curse horizon: Lufinitehhorizo off-policy estimation. In Advances in Neural Information Processing Systems, pages 3356-5366, 2018.
[39] Qinghua Liu, Praneeth Netrapalli, Csaba Szepesvari, and Chi Jin. Optimistic mle: A generic model-based algorithm for partially observable sequential decision making. In Proceedings of the 55th Annual ACM Symposium on Theory of Computing, pages 363-376, 2023.
[40] Vincent Liu, Prabhat Nagarajan, Andrew Patterson, and Martha White. When is offline policy selection sample efficient for reinforcement learning? arXiv preprint arXiv:2312.02355. 2023.
[41] Kohei Miyaguchi. Almost hyperparameter-ffee hyperparameter selection framework for offline policy evaluation. In AAAI Conference on Artificial Intelligence, 2022.
[42] Wenlong Mou, Ashwin Pananjady, and Martin J Wainwright. Optimal oracle inequalities for projected fixed-point equations, with applications to policy evaluation. Mathematics of Operations Research, 48(4))2308-2336, 2023.
[43] Alfred Müller. Integral probability metrics and their generating classes of functions. Advances in Applied Probability, 1997.
[44] Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections. Advances in Neural Information Processing Systems, 32, 2019.
[45] Anusha Nagabandi, Gregory Kahn, Ronald S Fearing, and Sergey Levine. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. In 2018 IEEE international conference on robotics and automation (ICRA), pages 7559-7566. IEEE, 2018.
[46] Allen Nie, Yannis Flet-Berliac, Deon Jordan, William Steenhergen, and Emma Brunskill. Data- efficient pipeline for offline reinforcement learning with limited data. Advances in Neural Information Processing Systems, 35: 148811141482, 2022.
[47] Tom Le Paine, Cosmin Paduraru, Andrea Michi, Caglar Gulcehre, Konrad Zolna, Alexander Novikov, Ziyu Wang, and Nando de Freitas. Hyperparameter selection for offline reinforcement learning. arXiv preprint arXiv:2007.09053 2020.
[48] Juan C Perdomo, Akshay Krishnamurthy, Peter Bartlett, and Sham Kakade. A complete characterization of linear estimators for offline policy evaluation. Journal of Machine Learning Research, 14(284):11500 2023.
[49] Doina Precup, Richard S Sutton, and Satinder P Singh. Eligibility traces for off-policy policy evaluation. In Proceedings of the Seventeenth International Conference on Machine Learning, pages 759-766, 2000.
[50] Touqir Sajed, Wesley Chung, and Martha White. High-confidence error estimates for learned value functions. arXiv preprint 1rXiv:1808.0911 09127, 2018
[51] Yi Su, Pavithra Srinath, and Akshay Krishnamurthy Adaptive estimator selection for off-policy evaluation. In International Conference on Machine Learning, pages
PMLR, 2020.
[52] Wen Sun, Nan Jiang, Akshay Krishnamurthy, Alekh Agarwal, and John Langford. Model- based RL. in Contextual Decision Processes: PAC bounds and Exponential Improvements over Model-free Approaches. In Conference Learning Theory, 2019.
13

---

[53] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[54] Shengpu Tang and Jenna Wiens. Model selection for offline reinforcement learning: Practical considerations for healthcare settings. In Machine Learning for Healthcare Conference, pages 2-35. PMLR, 2021.
[55] Philip Thomas and Emma Brunskill. Data-Efficient Off-Policy Policy Evaluation for Reinforce- ment Learning. In Proceedings of the 33rd International Conference on Machine Learning. 2016.
[56] Takuma Udagawa, Haruka Kiyohara. Yusuke Narita, Yuta Saito, and Kei Tateno. Policy- adaptive estimator selection for off- off-policy evaluation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 10025-10033, 2023.
[57] Masatoshi Uehara, Jiawei Huang, and Nan Jiang. Minimax Weight and Q-Function Learning for Off-Policy Evaluation. Proceedings of the 37th International Conference on Machine Learning, pages 1023-1032, 2020.
[58] Masatoshi Uchara, Xuezhou Zhang, and Wen Sun. Representation learning for online and offline in low-rank mdps. arXiv preprint arXiv:2110.04652, 2021.
[59] Claas A Voelcker, Arash Ahmadian, Romina Abachi, Igor Gilitschenski, and Amir-massoud Farahmand. X-ac: Learning latent decision-aware models for reinforcement learning in continu- ous state-spaces. arXiv preprint arXiv:2306. 17366, 2023.
[60] Cameron Voloshin, Hoang M Le. Nan Jiang, and Yisong Yue. Empirical study of off-policy policy evaluation for reinforcement learning. arXiv preprint arXiv:1911.0688. 2019.
[61] Cameron Voloshin, Nan Jiang, and Yisong Yue. Minimax model learning. In International Conference Artificial Intelligence and Statistics, pages 1612-1620. PMLR, 2021.
[62] Tengyang Xie and Nan Jiang. Batch value-function approximation with only realizability. In International Conference on Machine Learning, pages 11404-1143.. PMLR, 2021.
[63] Tengyang Xie, Ching-An Cheng, Nan Jiang, Paul Mineiro, and Alekh Agarwal. Bellman- consistent pessimism for offline reinforcement learning. arXiv preprint 1rXiv:2106.06926. 2021.
[64] Siyuan Zhang and Nan Jiang. Towards hyperparameter-ffee policy selection for offline rein- forcement learning. Advances Neural Information Processing Systems. 12875 2021.
[65] Joshua P Zitovsky, Daniel De Marchi, Rishabh Agarwal, and Michael Rene Kosorok. Revisiting bellman errors for offline model selection. International Conference on Machine Learning, pages 43369 43406. PMLR, 2023.
14

---

A Other Related Works
Adaptivity Guarantees for Offline RL Model Selection. Our theoretical guarantees state that the selected candidate function/model provides J (T)) estimate that is close to the groundtruth, as if the algorithm knew which candidate is correct. This can be viewed as a form of adaptivity guarantees, which is the goal of several theoretical works on model selection. Su et al. [5T] and Udagawa et al. (56) study adaptive model selection for OPE, but their approaches crucially rely the importance sampling estimator, which we do not consider due to the focus on long-horizon tasks. Lee et al.
[3 (who build on and improve upon the earlier works of Farahmand and Szepesvári [15] and Jiang et al. (24l) study model-selection of value functions, with a focus on the double-sampling issue and its relationship with Bellman completeness; these are also the very core consideration in our theoretical derivations. However, their approach treats the OPE instances (such as FQE) in a less black-box manner, and the goal is to select the "right" function class for the OPE algorithm (instead of directly selecting final output function) from nested series of classes, one of which is assumed to be Dellman-complett (6] [O] and have low generalization errors f This makes their approach less widely applicable than ours, and we cannot empirically evaluate their algorithm in our protocol as a consequence. Nevertheless, their results provide an interesting alternative and valuable insights to the model-selection problem, which are "complementary" to the BVFT line of work [62 [4] that we further develop. Similarly, Miyaguchi [41 considers the selection of Bellman operators, which can be instantiated as regression using different function classes, making their approach similar to Lee et al.
33 in spirit.
Other Works on Model Selection. Apart from the above works and those already discussed in the main text, most related works are not concerned about new selection algorithms with theoretical guarantees or experiment protocol for OPE model selection (see Voloshin et al. [60], Kiyohara et al. 26 for experiment protocol and benehmarks of OPE itself). so their focus is diffeerntt and often provides insights complementary to our work. For example, Nie et al. 46 discuss data splitting in offline model selection; this is question we avoid by assuming a fixed holdout dataset for OPE model selection. Tang and Wiens [54] compare importance sampling methods and FQE and conclude that FQE is more effective (which echos with Paine et al. (47)), but does not provide provable methods for selecting the hyperparameters of FQE, especially the choice of function approximation. Kumar et al. (88) provide a pipeline for offline RL that includes hyperparameter selection as a component, but the recommendations are heuristics specialized to algorithms such as CQL [27].
Debate on Bellman Error as a Proxy. Most of the selectors we consider estimate some variants of the Bellman error. Regarding this, Fujimoto et al. [16] challenge the idea of using Bellman errors for model selection due to their surrogacy and poor correlation with the actual objective. Despite the valid criticisms, there are no clear alternatives that address the pain points of Bellman errors, and the poor performance is often due to lack of data coverage, which makes the task fundamentally difficult for any algorithm. We still believe that Bellman-ent ciimann-error-lkee objectives (defined in a broad sense, which includes our LSTDD-ournammet) are promising for model selection, and the improvement on OPE error (which is what we report in experiments) is the right goal to pursue instead of correlation (which we know could be poor due to the surrogate nature of Bellman errors).
Data Coverage in Model Selection. As mentioned above and demonstrated in our experiments, the lack of data coverage is key factor that determines the difficulty of the selection tasks. Lee et al. 32) propose feature selection algorithms for offline contextual bandits that account for the different coverage effects of candidate features, but is unclear how to extend the ideas to MDPs. On a related note, ideas from offline RL training, such as version-space-bll space-based pessimism [63] can also be incorporated in our method. This will unlikely improve the accuracy of OPE itself, but may be helpful if we measure performance by how OPE can eventually lead to successful selection of performant policies, which we leave for future investigation.
Experimental Protocol. The closest to our experimental protocol is the work of Sajed et al. [50] who also uses Monte-Carle rollouts to produce unbiased estimates of value functions on individual states. Other than this point, their work is largely orthogonal to ours as they focus on establishing high- confidence bounds for the estimated value, assuming the simulator is the groundtruth environment,
'Given the relationship between Bellman-comme coiimaan-cmmltteness and bisimulation abstractions 10 Proposition 9). their setting is a natural generalization of Jiang et al. [24)ss 's setting of selecting a bisimulation from nested series of state abstractions.
15

---

whereas we draw Monte-Carlo trajectories from different simulators to facilitate the problem.
B Details of Model-based Selectors
Here we expand Section 4 on the model- basel setting, i.e., choosing model from (M, jig[m]- This a practical scenario when we have structural knowledge of the system dynamics and can build reasonable simulators, but simulators of complex real-world systems will likely have many design choices and knobs that cannot be set from prior knowledge alone. In some sense, the task is not very different from system identification in control and model learning in model-based RL, except that (1) we focus on a finite and small number of plausible models, instead of rich and continuous hypothesis class, and (2) the ultimate goal is to perform accurate OPE, and learning the model is only an intermediate step.
Existing Methods. Given the close relation to model learning, a natural approach is to simply minimize the model prediction loss [2] candidate model M scored by
E ((sss))))pp..sss)aaaaasa)s))))). (d(ss, 8)],
(12)
where in the data and generated according to the real dynamics (*(-|s, , a), and & is generated from the candidate model M's dynamics P. d(::-) is distance metric that measures the difference between states.
45 18 the method has major caveats: first, the distance metric d):.,) is design choice. When state is represented as a vealev vector, natural to the lg distance d(-,)). which changes if we simply normalize/rescale the coordinates. Second, the metric is biased for stochastic environments as discussed in prior works 59], which we also demonstrate in the experiment section (Section 6 essentially this is version of the double-sampling issue but for the model- setting [5). As minimal counterexample, suppose S C R² and we focus the transition distribution from particular (s, a) pair where groundtruth is uniform over points (+1, +1). The expected loss of the groundtruth model itself, in Eq. (2). is 1 22/2, this higher than wrong model that always predicts 0 deterministically, which yields loss of 22.
Despite its wide use and simplicity 17
There are alternative methods that address these issues. For example, in the theoretical literature, MLE losses are commonly used, i.e., [log P(s'ls, a)] [ 58 39), which avoids d(.,) .) and works properly for stochastic MDPs by effectively measuring the KL divergence between P*(-|s, a) and P(-ls, a). Unfortunately, most complex simulators do not provide explicit probabilities P(s' a). making it difficult to use in practice. Moreover, when the support Pa (-|s, a) is not fully covered by P(-1s, a), the loss can become degenerate.
To address these issues, we propose to estimate the Bellman error [q, P. where Q = Q3 As discussed earlier, this objective suffers the double-sampling issue in the model-free setting, which we show can be addressed when we have access to candidate models {M1,.... Mm that contains the true dynamics M Moreover, the Bellman error 1Qh, (8,a) (T" (7, =
y (Egr [Q, m)] Ex- [s, m)]l,
which can be viewed P(-|s, a) under Qc π) as a discriminator. IPM is also a popular choice of model learning objective in theory [52 611. and the Bellman error provides a natural discriminator relevant for the ultimate task of interest, namely OPE.
IPM loss 43 that measures the divergence between P*(+,s, a) and
B.1 Regression-based Selector
Recall that the difficulty in estimating the Bellman error E,[(Q, TTQ) is the uncertainty in T To overcome this, we leverage the following observation from Antos et al. (6, where for any 1:SxA->
THE arg min mig g.SxA=B
- - -
(13)
which shows that we can estimate T"Q, by solving sample-based version of the above regression problem with = Q. Statistically, however, cannot afford to minimize the objective over
16

---

all possible functions 9; we can only search over limited set G3 that ideally captures the target T=Q,-- Crucially, in the model-based setting we can generate such set directly from the candidates (M)iemj
Proposition 1. Let 9. (TM, Qi [m]]. Then if M E (Mi)eelm) it follows that T=Q; Th Q g.
The constructed G. ensures that regression is statistically tractable given its small cardinality, [Gal m. To select Qi we choose Qi with the smallest loss defined as follows:
1. arg yeg (pl(g(s, -r-?Q((s',=))
The loss Qi is ED(( a) - (s(s.)))
The 2nd step follows from Zitovsky et al. [65]. Alternatively, we can also use the min value of Eq. (3 (instead of the argmin function) to correct for the bias in TD-squared (Fq.22) []: see Liu et for another related variant. These approaches share similar theoretical guarantees under standard
analyses [62 63 and we only state the guarantee for the Zitovsky et al. [65] version below, but will
include both in the experiments ("mb Zitovsky et al." and "mb_ Antos et al" in Figure
Theorem 3. Let C == F
(s.a)
For Qi that minimizes ED a) (s(8,a)) have p(s.a)
w.p. 2 1 6,
V
152 C log 4m 8
[Tx Edo [Q)(8, S
(14)
n
B.2 Sign-flip Average Bellman Error
We now present another selector that leverages the information of (TM, j [m]]
different manner. Instead of measuring the squared Bellman error, we can also measure the absolute error, which can be written as (some (s, a) argument to functions are omitted):
E 11 TM-Qill
E2 (aggiiggniii Q2(s, ())
= E2 (sgn(( Qi - (()))))((((((()))))) - (ap0:(ss))----000000 -1-7Q((8,,))
S max F 3E9.
-
- -s
(15)
Here, the 9i from Proposition D inducess a set of sign functions sQn which includes Qi Qi-ii0 TT Qi. that will negate any negative TD errors. The guarantee is follows:
Theorem 4. Let Qj be the minimizer of the empirical estimate of Eq. 15 and C = maxx, p(s.a)
(s.a)
W.p. > 5.
log(2m/6))
J(x) Edo (((ss 54.C2 C., Vmmax V
7
C
Proofs
C.1 Proof of Theorem
Proof. Define the following loss vectors,
{(0) := A0 - E Rdd
?(0) Ao b E Rd
and recall that we select as the estimator
= arg min 100)
AE
17

---

Since 6* = =b"b. we can write the desired bound as function of ((0) as follows,
||QT() (.) x5 (-)01l
|1 (-) 0*) ||x0
(1)) '(A0 - bi)l
(+)AA (()A-)(0))l>>
max (8, (8, (a)a)(((())
8,0
max
| (s, a) ())12)))))))))) || 1le(0)l22
5,a
$
d ||A-1| |1(
Next, we control the (0) term. In the sequel we will establish via concentration that
log (2|
(e(0) - [(0) Estat == 3 max Rmax. B²}²
AA E O
(16)
n
Then, we have that
|xx - 11(00]1oo + Estat
|10(0))[lo + Estat
- ||((0)) + 2 Estat
2 Estat,
where we recall that @ = arg mingee [|((0)] 1lo in the second inequality, and that A0 = in the last line. Combining the above, we obtain
l|QT - Φ T - 2ddB.
||A "|1 Estat
log O
= 6vd. ||- (maxx B²}²
n
as desired. We now establish the concentration result of Equation (16).
Concentration results. For j E [d]. let (5) E IR refer to the j entry of the vector. For any (s, a, s') and define
BT (s, = o (s. (s. 70
T (s', m)0 r(s.a)
Recall that (lo(s, 2)l s Bg for all (s. a) and that ||0||, < Be for all E O. We have that, for all
j E [d], 8 O and s, E S A we have that (y a) a, s'; is bounded, since:
(s,( a)0 76 (s', v)) rss.a))
s | a)mmm (100(,, a) 10112 + 7llo(s' ())66''=)1221000)) |]]|||0||2 + Rmax)
s max (max/l0(s,00) all 8,0
max]lo(s a) + s.
+ Rmax
s,
$ (1 + 7) + Rmax B
< max BB, Rumxx²
Thus, from Hoeffding's 's inequality and union bound, we have that for all j E [d] and 0 E
Ep (s, s': 0)] Ep O0' (s, a) (s, s' [)] 3 max B Rmax 12 2 2log(d(0)5-1)
Estats
with probability at least - 8. As result, can write
|e(o] -(0) = E, [o((s, (ØT (x, a)0 7a (s', - r(s, a))] - E, (T (s, - 70 T ()', - r(s,a))]
F, (0(s, a) B a, 0)] (, a) (s,a,,,))) (Ba0
1 Estat bo
< Estat-
This concludes the proof.
18

---

C.2
Proof of Theorem 2
Proof. We first note that the proposed algorithm equivalent the following tournament procedure:
Vi E [m].j * ;
Define (i a) == [Qi(s, (,ls,,)),s(ssl and associated As, matrix and P vector
(Eq.
Define &a Acjei - bis E R²
Pick arg minke(() maxjzill Il 1|x
Let it E [m]] denote the index of Q in the enumeration of Q. We start with the upper bound
Jar- (T)) - Ex [Q; x)-l = dd IQ (s, Egnd, (Q, x)]l S 11 - 22
Let lij == Aiyei - bi.j denote the population loss. We recall the concentration result from Equation (T6), which, for any fixed and j, implies:
log
||e, Eoo I Estat 3 max Rmax 12
n
with probability at least - 5. This further implies || - bound over all (i.j) where either or j equal i*. this implies that
llé, S Estat- Taking a union
log (4dmm
||e,, 1o0 I Estat = 3-max Bsg Rmax 12
v(i.j) E (mm) x {(+))((("*} x [m])
n
If i = i* then we are done. If not, then there exists a comparison in the tournament where i = and
j i* For these features di,50 (8, a) [Q;(s, Qu- (s, T, have:
(+) - ax = +1 (E2 - e1)llx 1100 A 1,," Ai, (e2 - e1)looo
max (s, a), Ai,-- 1., (e2 - -e1)
9,a
10 (s.a)1: || ||||| A2,5 (E2 - e1)ll2
8:0
Vdl
max 3,
(8, a)]l2
|| 112l| (ez e1)11o0
vd
max s,
d (s, a all
12l1 1o
Va
max 8,
c (8, a)ll
1121 10
vd
max 8,
d (s, alll || 1l (14 + Estat)
Va
max 8,0
114 |1a (s, a)ll
max jE(m](((r))
11e + Estat
v
max 8,a
0 (s, 1l
||47
max jE(m])(fr)
|ler. if oloo + Estat
S vd
max 5,
d (s, a 4)ll
||A l12
max jelm/(fe)
|1e + 26stat
5 22
max (lx (8, a) 2|2
7, S|
8.0
22/d
max maxx|10; (8, a) 5.0
(E(m (E(m)N(r* Jmin (A1,.) E stat
max
19

---

To conclude, we note that d = 2 in our application and that max,,a (s, (s, I 2V max Plugging in the value for Esaat. this gives final bound of
()(s, +
1
|J (T) da (Q;(s, r))l = 4V max
max E [m)) min (Ai,e-) Estat
log 1og(4dmmo-1)
24
max
max ie [m] | min (Ar.s.)
n
C.3
Proof of Theorem 3
We bound
J(T) Edd (((,m)
Edg.
Q" - [(s,a))
E Q (s( (Q" ()),0) - mx)
(=((sa)) [T*0T ((s,a) + (s.a)
E
y
[ (s, a) a()
Ed
(( (17=0)(s.0)-
C En
where the second line follows from Bellman flow. Now we consider the term under the square root, and T6 = arg min mingegg Egg Q)
E ([Tr (s. - 0(s,a)))
(Q(,))))((
s 52.B, Bn [[T (s. (
@g(s. ()) x) +2 E, ( @(5 (T1)
(T2)
We consider each term above individually. (T1) the regression error between ğo and the population regression solution TQQ, which we can control using well-established bounds. The second term
(T2) measure how close the Q-value to estimated Bellman backup. To bound these two terms we utilize the following results. The first controls the eтror between the squared-loss minimizer JQ and the population solution T*Q, and adapted from [62]
Lemma 1 (Lemma 9 from 62 Suppose that we have (g) S Vinax for all g E GQ and Q Q and define
go = arg min ED [g(s, - 7Q(s' T))² gEGQ
Then with probability at least 1 8, for all i E [m]] we have
16V/2 max log
2m
Ey -
Freg
n
The second controls the error of estimating the objective for choosing z from finite samples, and proof is included at the end of this section.
Lemma 2 (Objective estimation error). Suppose that we have |[gl < Vmaa for all E GQ and Q E Q. Then with probability at least for all g E GQ and Q have
Eg - Q(x -({, Ep [o(s, a) -ss
max
2
Ep
(g(s,a) - Q(s a))
3
E, [g(s, a) - -
2m 3V2 log b
max
= Eobj.
2
20

---

Using Lemma we directly obtain that with probability at 1 - 6,
(TI) S reg
By leveraging Lemma
2 we have that with prohability at least 1 - 8,
(T2) Ez (Dg(s.a) 2(3)
(īa(s.a) -(, ) 21
S 2 Eobj + ED
$ Eobj
+ ED
(0 (8, (s(s
< 4 Eobj
+ B
(0q- (s, (s.
Cobj +3 y
(āor (8, -T"
< 4 Eobj + 3- Ereg
where in the first inequality we apply Lemma[[(by lower bounding the LHS with the first expression in the max); in the second use the Q-value realizability assumption Q" E Q with the fact that Q the minimizer of the empirical objective; and in the third we again apply Lemma 2 (now lower bounding the LHS with the second expression in the max). Then we use the identity that Q" =TQ", and apply the squared-loss regression guarantee. The bounds for (T1) and (T2) mean that
Ey (s.)
s 8(Eob) + eleg),
resulting in the final estimation bound of
J(r) Fedo [((,
Cx Rp (()))]( 2(s,aa))
5 1 1-
$ 1- v 8.C. (Eabj + Eiez)
V Vinax
152 C= loog 2m
which holds with probability at least 1 - 26.
Proof of Lemma 2
Observe that the random variable (g(s,a) ((s,((()))))) a))a E 1- 2
maxi V2 and
(g(ss,) Q(s,a)) En (0(s, a(s
V2 Vaax EE [g((s, Q(s.
Then, applying Bernstein's 's inequality with union bound, we have that, for any g E Go and Q E Q with probability at least -& 8.
En (g(s, a) a)) Ep [((s,)))Q(s,,)))
44 (o((s()) ((s,a)))
2m log
log
2m ar
max
n
n
4V2 V Ey (o(ss ((s, a))
log 2mz
V2 max
log( 2m
n
72
Ep
(g(8, a) Q(s,a))²
312 Max log 2m 5
2
21

---

Expanding the absolute value on the LHS and rearranging, this then implies that
3
log 201)
2 Ex -((s,,))(((sa)))))
Q(, a))
I ED -((s.))))
+
max
3 ED (o(s,aa 5 Ex -
3V
max
log 2m
n
Combining these statements completes the proof.
C.4 Proof of Theorem
We bound
((T) Edd (({,T)
Exto
a=(ssa)
Ed Q (s, - ~
(TT(a,)) (TQ [ (s.a)
Ex
EddT [[T (8,00 - [(s.0))
1
C
Ex (s,0) -
< max Ry sgn ((s,a) g(s, a(.,a))
m1)]
3E9s
By assumption, max I Vmax. and similarly maxgegQ |1 Vmax for all Q Q. Then for any Q E Q and E 9Q and (s, a) E S x A and r E [0, Rmmx))
sgn(Q(s, a) - g(s,a)))( (Q(ss a) - -r''s's', m)) E e Vimxl:
and, using Hoeffding's s inequality, we have for all Q E Q and E Go that, with probability at least 1 8,
Ep sgn -g(,,)))ss,) - -r [m)]
log( 2m
- En sgn (((s. - g(s, (Q(s.))))(() -r x) < 2V 2/max
== Eobj-
Then using this concentration in the last line of the previous block.
((x) - Esto [0(,, m))
< max ED sgn (Q(s, - 0(s, -r ) + Eabj
gEsig
< max
g(s, (s.a) ~Q" (s', m))] + Sobj
gESign
< max gE9gm
|yn ( (8, a) g(8, a) (3 - (2" [[',,))]] + 2 Eabj
max Eylsgn Q² (s, a) g(s, g(s. -
+ Eobj
Eg (8, - g(s, ((= - (x a)) + 2- Eabj
max 9E9q=
2 Eobj.
where in the first and third inequalities we apply the above concentration inequality, and in the second inequality we use the fact that Q is the minimizer of the empirical objective, i.e.,
Q arg min max En (n(sgn(Q(s, a) g(s, a)) QEO gEso
((s(,=)))
Combining the above inequalities, we obtain the theorem statement,
J(z) Edo [((s, m) S C% Sobj-
22

---

C,5 Further Discussion on Comparison to BVFT and the Coverage Assumptions
Besides the difference in rates, the guarantees for ASTTD-Tounameet and BVFT differ in their coverage assumptions, which also differ from those for the model-based selectors TThhorremsBand 4. In fact, Theorems 3 and 4 use the standard "concentrability coefficient" C that is widely adopted in offline RL theory 16 [10]. In contrast, TASTD--Tummmen'' ' coverage parameter is maxie(m|\M()) Mminn "" which is a highly specialized notion of coverage inherited from that
of LSTSD (ini (A) from Theorem D. Similarly, the coverage parameter of BVFT is also notion of coverage (called "aggregated concentrability" [62] 19) specialized to state abstractions. As far as we know there no definitive relationship between the coverage parameters of LSTD-Touraamen and
BVFT and neither dominates the other. Furthermore, these non-standard, highly algorithm-specific coverage definitions are common situation for algorithms that only require realizability (instead of Bellman-compllleness)) as their expressivity condition, as discussed in Jiang and Xie [23]
Problems in miin (A) LSTDQ's Coverage Assumption As mentioned above, the coverage condition of LSTD- Tournament is inherited from its "base" algorithm. i.e., the mmin (A) term in LSTDQ. Despite its wide use in the analyses of LSTDQ algorithms 30 3 48 we discover several problems with this definition that warrants further investigation. First, the quantity is not invariant to reparameterization, in the sense that if we scale the linear features é with a constant (and scale O accordingly), min (A) will also change despite that the estimation problem essentially remains the same. Second, if Φ has linearly dependent coordinates, miin (A) can be 0 while this says nothing about the quality of data (which could have perfect coverage); this is usually not a concern in LSTDQ analysis since we can always pre-process the features, but for its application in LSTD-Tournament the lack of linear independence can happen if we have nearly identical candidate functions Q, z Qj- Last but not least, we lack good understanding of when min is well-behaved outside very special cases; for example, it is well-known that fully on- on-policy data (in the sense that μ is invariant to the transition dynamies under π) implies nower-bounded amin (A). However, as soon as 1 is off-policy, there lacks general characterization of what kind of μ helps lower-bound miin minn(A). For example, when concentrability coefficient 5% well bounded, it is unclear whether that implies lower-bounded mmin (A).
In this project we have explored ways to mitigate these issues, such as defining coverage as Tiin (m) (A) to avoid the linear-dependence issue. However, none of the resolutions we tried solve all problems elegantly. This question is, in fact, quite orthogonal to the central message of our work and of separate interest in the fundamental theory of RL, and speaks to how algorithms like LSTDQ are seriously indetrinvestigated. In fact, coverage in state abstractions has been similarly under-investigated and has only seen interesting progress recently [99]. We wish to investigate the question for LSTDQ in the future, and any progress there can be directly inherited to improve the analysis of
D Experiment Details
D.1 Environment Setup: Noise and State Resetting
State Resetting. Monte-Carde rollouts for Q-value estimation rely on the ability to (re)set the simulator to particular state from the offline dataset. To the best of our knowledge, Mujoco environment does not natively support state resetting, and assigning values to the observation vector does not really change the underlying state. However, state resetting can still be implemented by manually assigning the values of the position vector qpos and the velocity vector quel.
Noise. As mentioned in Section 6 we add noise to Hopper to create more challenging stochastic environments and create model selection tasks where candidate simulators have different levels of stochasticity. Here we provide the details about how we inject randomness into the deterministic dynamics of Hopper. Mujoco engine realizes one-step transition by leveraging mjData. {ctrl, afrc_applied! objects [33, where mjData. ctrl corresponds to the action taken by our agent, and mjData. SegDaaaaaqff__aaae bations in the joint space and Cartesian coordinates, respectively. To inject randomness into the transition at noise level of a, we first sample an isotropic Gaussian noise with variance o² as the stochastic force in Data xfre [:3] upon each transition, which jointly determines the next state with the input action mjtaa ctrl, leaving the joint data mjData. qfrc intact.
xfrc_applied} are the user-defined pertur-
23

---

D.2 Experiment Settings
MF/MB.G/N. The settings of different experiments are summarized in Table ! We first run DDPG in the environment of g = -30, = 32, and obtain 15 deterministic policies {T0:14) from the checkpoints. The first are used as target policies MF.G/N experiments, and MB.G/N use fewer due to the high computational cost. For the main results (Section 6.1). the choice of M is usually the two ends plus the middle point of the grid (Mg or M,). The corresponding behavior policy is an epsilon-greeds version of one of the target policies, denoted as Ti which takes the deterministic action of (5) with probability 0.7, and add a unit-variance Gaussian noise to (T) with the remaining 0.3 probability.
OfN.GN. In the above setup, the behavior and the target policies all stem from the same DDPG training procedure. While these policies still have significant differences (see Figure 2 the distribution shift is relatively mild. For the data coverage experiments (ocction66.)., we prepare a different set of behavior policies that intentionally offer poor coverage: these policies, denoted as poor are obtained by running DDPG with a different neural architecture (than the one used for generating 70:14) in a different environment of g - 60, 100. We also provide the parallel of our main experiments Figures and under these behavior policies with poor coverage in Appendix E.1
MET.G. This experiment is for data coverage (Section 6.3
where 'D is mixture of two datasets, one sampled from Ty (which is the sole target policy being considered) and one from poor that has poor coverage. They are mixed together under different ratios as explained in Section 6.3
Groundtruth Model M* and
Target Policies
Gravity
Noise Level o
Behavior Policy mb
MEG
LIN(-51, -9,11,
100
{(M (MM,, (0, 7, 14)]
To:9
MEN
-30
LIN(10, 100, 15)
[M, (,) (0,7,14)
To:9
MB.G
LIN(-36)
-24, 22
100
{[M,, m mM (0,2,4)
To:5
MB.N
-30
LIN(10, 100,5)
{(M,, ff), )
(0,2,4)
To:5
MF.OFG.
-9,15 15)
100
{(M, m.
poor
) (0,7.14))
To:9
MFOFEN
-30
LIN 100, 15)
{MM,, T. boor
(0,7,11,4
To:99)
MET.G
LIN(-51,-9,11
100
{(Me, T8 क.
",iE 10)
{TH)
Table I: Details of experiment settings. LIN(a, b, n) (per numpy convention) refers to the arithmetic sequence with n elements, starting from and ending in (e.g. LIN(O, {0, 0.4, 0.6, 0.8, 1.0)).
D.3 Computational Resources
We here provide brief discussion of the amount of computation used produce the results. The major cost is in Q-caching, i.e., rolling out trajectories. The cost consists of two parts: environment simulation steps (CPU-intensive) and neural-net inference for policy calls (GPU-intensive). The main experiment on MF.G, took nearly a week on 4090 PC. The runtimes of MEN, MB.G, MB.N are comparable MF.G.
E
Additional Experiment Results
Subgrid Studies. Figures and show more complete results for investigating the sensitivity to misspecification and gaps Section 6.2 across settings (good/poor coverage and gravity/noise grid).
24

---

MI.G: c=100.0
MEN: d=35.0
MFOFFF:: a=100.0
OFF
20
2
OPE Ereor
15.
OPE Error
OPE Erro
OPE Ermr
10
2
0-
20
20 Noise Gap
20
20
Gravity Gap
Gravity Gap
Noise Gap
random
square
mf, avg. beliman, orror
M toumament
mf_bvit
Figure 6: Subgrid studies for gaps. Plot MEN identical to Figure
MFG: gm-51, e=100.0
MFN:: gm-30, 1=10.0
WFOFFG:.gm55..0100
OFF N: gm-30, am10.0
OPE Error
OPE Error
4
OPE Error
20
OPE Enror
30
15
0
0
Misspecification Level
Misspecificatin
Misspedification Level
Misspecification
random
mf. td square
avo bellman error
mf Istdo toumament
mf. bvit
Figure 7: Subgrid studies for misspecificattion Plot MEN is identical to Figure M.
shows more complete results for the data coverage experiment in Data Coverage. Figure
Section 6.3
including more choices of M
91-51. 0-100.0
1-100.0
gm.9. 100.0.0
20
random
square
OPE Error
OPE Error
OPE Error
30
mf ava_ beliman, mf. Istda toumament
20
5
5-
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Ratio of Target Data A
Ratio of Target Data A
Ratio of Target Data A
Figure 8: Data coverage results. The left figure is identical to Figure 5
F. Poor Coverage Results
We now show the counterpart of our model-free main results (Figure 3) under behavior policies that offer poor coverage. This makes the problem very challenging and no single algorithm have strong performance across the board. For example, naïve model-based demonstrate strong performance in ME.OFE.G (top row of Figure 99) and resilience to poor coverage, while still suffers catastrophic failures in ME.OFEN. While also has worse- whaathandononnn performance in one of the environments in M.OFFEG.
generally is more reliable than other methods, it
25

---

MI. DF::FF::: 5-100.0
OFF:G: a=100.0
MEOFF:: a=100.0
15.0
30
25
random
7.5
[d. square
belimen
mf. Istdq toumament
bvft
OPE Enor
1000 2000
3000
1000
2000
3000
1000
2000
3000
mb_
Sample Size
Sample Size
Sample Size
MF.OFFN: g-
MIF:FF.N g- 5555.0
MF.FFFN: g-100.0
Figure 9: Model-free se- lection results under be- havior policies with poor coverage (ME.FF.G/N).
6-
2-
4-
1-
2-
1000
2000
3000
1000
2000
3000
1000
2000
3000
Sample Size
Sample Size
Sample Size
E.2
LSTDQ Family
As mentioned at the end of Section 3 our LSSD--Toummmen can have several variants depending on how design and transform the linear features. Here compare 3 of them in Figure TO The method in all other figures corresponds to the "normalized _dff" version.
Vanilla: Φ., Qg1
Normalized: Φ = [Q1/ Qj/c/), where
VV (s.a))u (s. a)]
normalizes the
discriminators to unit variance on the data distribution. In practice these variance parameters are estimated from data.
Normalized diff: Φi.j = [Q:/;, c. (Q, - -2))/ where & and C5 performs normalization in the same way above.
26

---

MFG:: gm-51, a=100.0
MF.:: gm-30 a=100.0
MF.G: 00000000000
12.5
12
30
OPE Eroor
10.0
OPE Error
OPE Error
25
20
5.0
15
2.5
10
1000
2000
3000
1000
2000
3000
1000
2000
3000
Sample Size
Sample Size
Sample Size
MEN:: gm-30, 1=10.0
MFNN: g=-30, c=55.0
MFN: g=-30, 1=100.0
1.8
OPE Error
OPE Eror
1.6
3.0
1.4-
2.5
1.2
1.0
1000
2000
3000
1000
2000
3000
1000
2000 3000
Sample
Sample Size
MF.OF.:: g=:51, 1=100.0
MFOFFG:: q=-30, a=100.0
MF.OFF.:: g=-9, c=100.0
20
18
12
varilla
16
Emer
OPE Error
lized
14
OPE
10
1000
2000
2000
1000
2000
3000
1000 2000 3000
Sample Size
Sample Size
Sample Size
MF.DFFN: 1010.0
Mr.DFEN: 5=55.0
MF OFFN: q=-30, a=100.0
2.0
OPE Enoor
OPE Error
OPE Error
1.5
1
1000
2000
3000
1000
2000
3000
1000
2000
3000
Sample
Sample
Size
Figure 10: Comparison of variants of LSTD-TToumament.
27

---

E.3 ((1) Rollouts
In our experiment design, we use fairly significant number rollouts 128 to ensure relatively ac- curate estimation of the Q-values. However, for the average Bellman error and the SSTD-Toumammet algorithms, they enjoy convergence even when is a constant. For example, consider the average Bellman error:
ED(Q a) - - 7Q:6s' m)),
which is an estimation of (Q,(( - -s:''' Thanks to its linearity in Qi. replacing Qi with its few-rollout (or even single-rollout) Monte-Carlo estimates will leave the unbiasedness of the estimator intact, and Hoeffding's s inequality implies convergence as the sample size n [) increases, even when stays as a constant, which is an advantage compared to other methods. That said, in practice, having relatively large can still be useful as reduces the variance of each individual random variable that we average across D, and the effect can be significant when n is relatively small.
A similar but slightly more subtle version of this property also holds for LSTDD-Toumammnt. Take the vanilla version in Section example, we need to estimate
Ep(Qj ()(Qa())()))))))) a) - - m))]-
Again, we can replace Qj and Qi with their Monte-Carlo estimates, as long as the Monte-Carlo trajectories for Q, and Qj are independent. This naturally holds in our implementation when j # but is violated when j = since (s, a) and (s,s, a) will share the same set of random rollouts, leading to biases. A straightforward resolution is divide the Monte-Carke rolllouts into two sets, and (, Is.0 a) and (s, a) can use different sets when j * i. We empirically test this procedure in Figure 11 where the OPE errors of average Bellman error and different variants of LSSDD-Tounamen are plotted against the number of rollouts in each set (i.e., 1/2). While small number of rollouts can sometimes lead to reasonable performance, more rollouts are often useful in providing further variance reduction and hence more accurate estimations.
MFG: gm-51, 1=100.
MFG: gm-30, 1=100.
MFG: gm:9, m=100.0
20-
random
35
OPE Emrr
30
OPE Error
25
30
normal loed
normal parrmaized_dft
10
OPE
25
20
0
20
40
Rollouts
MF G: g- 1-100.0
MFG:: g- 90.30, d-100.0
MF.G: 9, 0-100.0
10
Noralized DIF Loss
Normalized DIf Loss
Normalized DIT Loss
8-
Emy1
Em
+
Env 10
6
Env4 Env 23
Eny
Erv
Env 14
2-
20
Rollouts
Roll outs
Rollouts
Figure 11: The effect of small rollouts in LSTTD-Tournamen methods. Sample size is fixed 3200 and only (the number of rollouts) varies. Since the rollouts are divided into two separate sets to ensure independence, each Q-value estimated using 1/2 rollouts in this experiment, which is shown on the x-axes. The top row shows the OPE error (i.c., final performance), whereas the bottom row shows the convergence of loss estimates.
28

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We claim to provide new algorithms (Sections and with theoretical
guarantees (Theorems 2 and 1 and experimental protocols (Section > that is instantiated
provide empirical results (Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
29

---

Answer: [Yes]
Justification: The standard assumptions that are often treated as part of the problem setup (e.g., OPE with i.i.d. data, realizable O-function/model set) are stated in Section 2 Other than those, our theorems only require data coverage assumptions, which are manifested as the coverage parameters that explicitly appear in the bounds. All proofs are provided in Appendix C
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Additional experiment details can be found in Appendix Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
30

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: The code is uploaded to the supplementary files. The data be generated from the simulators and the generation code is included.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https //nips. htzjps.ccpublic/guiie.. htwbbl/i/ggides/odeeeeeeeeeee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Our results are not about standard training but model selection in evaluation, and all compared algorithms are optimization-free and closed form, so the above aspects mostly do not apply. The closest is how the environments are set up and how the behav- ior/target policies are created, which can be found Appendix D. and D.2 Our new algorithm LSTD-Tourammnt has a few variants (and the choice can be viewed form of hyperparameter), which are tested and compared in Appendix E.2
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We use bootstrapping to calculate enror bars that reflect standard errors in the "Bootstrapping" paragraph Section 5.2
Guidelines:
The answer NA means that the paper does not include experiments.
31

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The computational resources used are discussed in Appendix D.3
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't It make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: Our work is foundational, focusing on theoretical development and simulation results. The topic of study, OPE, aims at providing estimation of new policies before deployment, which is beneficial to the safe use of ML in applications. Overall, we do not think the work presents any ethical concerns that worth highlighting.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
32

---

Justification: Our work is foundational, focusing on theoretical development and simulation results. The topic of study, OPE, aims at providing estimation of new policies before deployment, which is beneficial to the safe use of MI in applications. Overall, we do not think a discussion of broader impacts in the paper is necessary.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification:
We used the Gym Hopper environment, which cited the end of Section
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
33

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperssi thoode com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's ' creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Readme is included in the code zip file.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the datasct/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
34

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM for what should or should not be described.
35


Reviews:
[{"review_id": "IamjRWm95r", "reviewer": "Reviewer_Robe", "summary": "This paper attempts to address the problem of model selection for off-policy evaluation (OPE) algorithms. It introduces a hyperparameter-free model-free selector named LSTD-Tournament, derived from the LSTDQ algorithm, and provides an accompanying theoretical analysis. The paper also proposes several new model-based selectors and a new experimental protocol for evaluating OPE selectors. This protocol generates candidate Q-functions by perturbing the parameters of a ground-truth simulator. Experiments on the Gym-Hopper environment are presented to evaluate the proposed methods.", "questions": "1. Could you clarify the motivation behind the design of the new model-based selectors? What advantages were they hypothesized to have?\n2. Regarding the claim that the naive model-based selector fails in stochastic environments, could you elaborate on how Figures 2 and 3 demonstrate this? A more direct explanation in the text would be very helpful, as the visual evidence alone is not entirely clear.\n3. Could you comment on the generalizability of your results beyond the Hopper environment and DDPG-based policies?\n4. The LSTD-Tournament loss uses candidate functions $Q_k$ or the normalized difference to weigh the Bellman residual of $Q_i$. Could you provide a more concrete intuition for why this is a good design choice? Have you considered or experimented with other weighting functions to verify that using the candidate Q-functions themselves is critical to the method's success?", "limitations": "Yes.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strengths\n- **Relevant Problem Formulation**: The paper focuses on a significant and challenging problem in practical offline RL: how to select among different OPE estimators or their configurations. This is a critical step for reliable policy selection before deployment.\n- **Solid Theoretical Guarantees**: The LSTD-Tournament selector is supported by a clear theoretical analysis (Theorem 2), demonstrating a standard $n^{−1/2}$ convergence rate, a notable improvement over BVFT.\n- **An Attempt at a Better Protocol**: The proposed experimental protocol tries to solve a real problem with existing benchmarks that rely on unstable FQE instances. The idea of generating candidates from perturbed ground-truth models is an interesting direction for creating more controllable testbeds.\n## Weaknesses\n1. **Clarity and Organization**: One of the paper's primary weakness is its poor organization and clarity, which makes it extremely difficult to follow. The flow of information is chaotic and confusing.\n    - *Disruptive Forward References*: The text is littered with forward references that disrupt the reading flow and require the reader to jump to later parts of the paper (or the appendix) before the necessary context is provided. For instance:\n        - In the Preliminaries (Section 2), the text directs the reader to \"see Figure 5 for a visualization\", but Figure 5 is in the appendix and illustrates the experimental protocol from Section 5.\n        - Section 3.1 refers to \"e.g., \\mathcal{G} in Section 4\", a concept not yet explained.\n        - Section 5.2 (Computational Efficiency) tells the reader to \"see Section 6.2\" for an example of its application, jumping ahead of the entire experimental setup.\n    - *Vague Justification of Claims*: Some claims are not adequately explained. For example, when discussing the naive model-based baseline, the paper claims there is \"empirical evidence of this in Section 6 (see `mb_naive` in Figures 2 and 3)\" but provides no further interpretation. Looking at the plots, it is not immediately obvious how they serve as damning evidence; in several high-noise environments (e.g., top row of Figure 2), `mb_naive` appears to perform quite well or at least comparably to other methods.\n    - *Minor Typos*: There are some typos, such as \"textitunits\" in Section 5.1.\n2. **Misleading \"Hyperparameter-Free\" Claim and Unmotivated Design**: The paper's central claim of LSTD-Tournament being \"hyperparameter-free\" is questionable. The design of the feature map $\\phi(s,a)$ (e.g., $[Q_i, Q_j]^T$ or the normalized difference version) is a critical design choice with a significant impact on performance, effectively acting as a hyperparameter. In fact, the weighting function $Q_k(s,a)$ in Eq.10 can be any function of $(s,a)$, effectively introducing infinitely many degrees of freedom. The paper acknowledges variants (Section 3.2, Appendix E.2) but heuristically fixes one for the main experiments, which merely hides this degree of freedom rather than eliminating it. The ablation study in Appendix E.2 demonstrates that the performance of LSTD-Tournament is quite sensitive to choice of the weighting function. This core design choice lacks a strong intuitive motivation and is not properly justified through ablation studies comparing it with simpler alternatives, e.g., random functions of $(s,a)$.\n2. **Underdeveloped Model-Based Contribution**: The section on new model-based selectors (Section 4) feels extraneous and not well-motivated.\n    - *Poor Performance and High Cost*: These selectors are empirically shown to be outperformed by the simpler, more efficient, and more broadly applicable model-free LSTD-Tournament. They also have a higher computational cost ($O(m^2)$), making them less practical.\n    - *Lack of Motivation*: The paper offers little intuition for the design of these new selectors (especially the \"Sign-flip Selector\"). It is unclear what benefits they were intended to provide over existing approaches or the model-free methods.\n3. **Limited Diversity in Experiments**: The generalizability of the empirical findings is constrained by a lack of diversity in the experimental setup.\n    - *Single Environment*: All experiments are conducted exclusively on the Hopper-v4 environment. Although the authors introduce variations by modifying parameters like gravity and noise, the underlying task structure remain the same. Testing on a wider range of distinct environments would significantly bolster the claims of robustness.\n    - *Homogeneous Policies and Datasets*: The target policies are all generated from checkpoints of a single algorithm, DDPG. The behavior policies are also created in a uniform way, typically by adding Gaussian noise to a DDPG policy. A more realistic scenario would involve selecting between policies generated by different types of algorithms (e.g., CQL, IQL, TD3) and using offline datasets collected from varied sources, which this study does not cover.", "was_revised": true, "final_justification": "This paper presents a novel selector, LSTD-Tournament, for a significant problem in offline RL. The discussions with the authors were productive and helped clarify their contributions and my concerns. While the authors provided a clear explanation for the failure of the `mb_naive` baseline, several major issues remain unresolved.\n\nMy recommendation remains **Borderline Reject**. The paper contains a promising direction but is undermined by fundamental issues in its claims, theoretical framing, and presentation.\n\nHere is a summary of the unresolved issues that form the basis of my recommendation:\n\n* **Overstated \"Hyperparameter-Free\" Claim (High Importance):** This is the most critical unresolved issue. The authors' central claim that LSTD-Tournament is \"hyperparameter-free\" is not accurate. The choice to set the weighting function equal to the feature basis is a significant design decision that functions as an implicit, high-impact hyperparameter. The authors' defense rests on the \"on-policy boundedness\" property of this choice, but this property's relevance in the challenging off-policy setting is not well-established and does not preclude the potential superiority of other weighting functions. Ignoring this vast design space weakens the paper's core premise.\n\n* **Poor Clarity and Organization (High Importance):** The paper is exceptionally difficult to read. This concern was not adequately addressed and was echoed by other reviewers. The chaotic organization, disruptive forward references, and lack of clear motivation for key components make it a significant challenge for the reader to understand the work and appreciate its contributions.\n\n* **Disconnected Theory and Unmotivated Model-Based Section (Moderate Importance):** The theoretical guarantees for the model-free (Theorem 2) and model-based (Theorems 3 & 4) selectors are built on different, incomparable assumptions. This makes it impossible to theoretically assess if or when the model-based methods should outperform the model-free one. Given their poor empirical performance, this section feels like a collection of unmotivated and inefficient algorithms. The value of reporting these \"negative results\" is diminished because the analysis does not explain *why* these theoretically sounder methods failed.\n\nIn conclusion, while the LSTD-Tournament algorithm is interesting and the paper has valuable technical components, the combination of an overstated central claim, a disconnected theoretical narrative, and severe clarity issues prevent me from recommending acceptance. The paper requires substantial revision to address these foundational problems.", "rebuttal": "Thank you for your detailed and helpful feedback! We will refer to your listed weaknesses as W1-4 and questions as Q1-4.\n\n---\n\n**W2: Misleading \"Hyperparameter-Free\" Claim and Unmotivated Design**\n\nWe first address a potential **technical misunderstanding** before responding to your comments. \n\n> **”the weighting function $Q\\_k(s,a)$ in Eq.10 can be any function of $(s,a)$, effectively introducing infinitely many degrees of freedom.”**\n\nThis is **incorrect**. In particular, choosing any function, such as a purely random function of $(s,a)$ (more on this below), could lose the guarantees of Theorem 2. As we mentioned on L180, only “non-degenerate **linear transformations**” of $[Q\\_i, Q\\_j]$ enjoy the same kind of guarantees. **Choosing an arbitrary weighting function unrelated to $Q\\_k$ simply cannot be called a variant of LSTD-tournament.**\n\nWe hope this clarification eases your concerns. Below we respond to your other related comments, assuming you still stand by them after this clarification. As you have noted, we acknowledge this degree of freedom (**the linear transformation**) and provide a recommended choice (normalized diff). That said, there is a big difference between this kind of “small” hyperparameters vs. the “big” ones we mention in the introduction (e.g., neural architecture of FQE; L35). To put it simply, \n\n> Are we comfortable recommending neural nets of 2 hidden RELU layers with 100 nodes per layer whenever someone applies FQE in _any OPE problem_? **Of course not!**\n\n> Are we comfortable recommending the normalized diff version whenever someone applies LSTD-tournament to _any OPE model-selection problem_? **Yes! (see reasons below)**\n\n**lacks motivation for the design, and ablation in App E.2 demonstrates [sensitivity]**\n\nThere is good motivation for normalization. In particular, imagine if a wrong candidate $Q\\_k$ has enormous magnitude. When such a $Q\\_k$ serves as the weighting function, it can artificially blow up the loss of other $Q\\_i$, including the true Q-function, and the proof of our theoretical guarantee relies on true Q having a reasonably small loss. Therefore, controlling the magnitude of weighting functions via normalization is well-motivated. Taking difference is, indeed, more heuristic, but its use in RL to help with numerical stability is also well-documented [25,10].\n\nThe 3 variants do have different performance profiles in App E.2. However, compared to other baselines, the qualitative nature of their performances are not that different, and none of them strictly dominate the others. Given that their performance comparison is largely a tie, and there is good reason to prefer normalization (to protect against large Q) and difference (for numerical stability), we feel comfortable recommending the normalized diff version as a **universal choice**. Fixing such a choice, in our opinion, is _exactly eliminating the degree of freedom_ rather than “hiding it”. For example, BVFT also has a discretization error as a hyperparameter, and the authors provide a heuristic to pick it in [61], which we adopt in our implementation.\n\n---\n\n\n**Q4: why are $Q\\_k$ good weighting functions intuitively?** \n\nUsing $Q\\_k$ as weighting functions purely emerges from our theory. We don’t have more intuitive explanations other than the theory, which we don’t view as a negative; in fact, BVFT is just as counterintuitive as (if not much more than) LSTD-tournament.\n\nPopping up a level, most RL research follows an “empirical-first” methodology of “**design intuitive algorithms -> test empirically -> if works, find theoretical justifications**”, which is perhaps why the reviewer asks this question. In contrast, we start from derivations with strong theoretical guarantees (Theorem 2) as the basis of research. This methodology has the unique advantage of tapping into those “counterintuitive” ideas in the algorithmic space. If not for it, algorithms like BVFT will perhaps never be found. \n\n**W2: How about other weighting functions, such as “random functions of (s,a)”?**\n\nPerhaps surprisingly, if you use “random functions” as weights, the loss becomes degenerate. In particular, we can write Eq.10 as\n\n$$ \\max_{w \\in  \\mathcal{W}} |\\mathbb{E}\\_{\\mu}[w(s,a) (Q\\_i-\\mathcal{T}^\\pi Q\\_i)]|,$$\n\nAs you pointed out, LSTD-tournament uses $Q\\_k$ as the weighting function $w$. The most random way to generate $w$ is to generate an i.i.d. weight scalar for each $(s,a)$ in the dataset. Suppose this i.i.d. distribution has mean $C$. Now, because the randomness in weight is independent of everything else, the outside expectation will effectively first marginalize out such randomness, which means the expected loss is simply equal to\n\n$$|\\mathbb{E}\\_{\\mu}[C (Q\\_i-\\mathcal{T}^\\pi Q\\_i)]|.$$\n\nWhen $C \\ne 0$, this coincides with “average Bellman error” (L290) which we include in the experiments. It has poor theoretical properties: even if a candidate $Q$ has the wrong shape, you can always add a constant to it–which does not change its shape–to make its average Bellman error $0$. \n\nIf we want make the weights “nontrivially random”, we need to leverage information in $(s,a)$, e.g., generate a random function that is smooth over $S \\times A$ w.r.t. some metric. However, the choice of the metric and how exactly the function should be generated are “big hyperparameters” that are left as design choices to the user.\n\nMoreover, the sign-flip selector (App B.2) also fits this form, where $sgn(Q\\_i - g)$ serves as weighting functions. So that can also be viewed as a comparison where the weighting functions are chosen differently.\n\n---\n\n**Q2 \\& W1: explain the failure of naive\\_mb**\n\nAs mentioned on L203, naive\\_mb has a strong bias towards more deterministic systems. That is, **when the candidate models have varying degrees of stochasticity**, naive\\_mb tends to choose the least stochastic one. You observe that naive\\_mb performs well in the top row of Fig 2, and that’s because all candidate models in MF.G have the same degree of stochasticity. (There is a potential confusion: “MF.G” and “MF.N” are specified on L271; the value of $g$ and $\\sigma$ after “:” describes $M^\\star$, **not** “MF.G” or “MF.N”.) **It’s the relative difference in stochasticity among candidate models that matter, not the absolute level.** This also explains why naive\\_mb performs particularly well in the left plot of bottom role, since the groundtruth model has the least stochasticity and naive\\_mb’s bias happens to play in favor. \n\nWe believe this particular bias is well known to the theory community (since the underlying issue is a strong motivation for widely used tools such as Wasserstein distance) and thus omitted a detailed explanation. Given your feedback, however, we will add more discussion. The key is that for $\\\\\\| s’ - \\tilde{s}\\\\\\|$ in the loss (L120), $s’$ is from the data and $\\tilde{s}$ from the candidate model. When the candidate is the true model, $s’$ and $\\tilde{s}$ are iid, but **their randomnesses are independent and “out of sync”**, which causes the loss to be non-zero and scale with the inherent randomness of the distributions. As a minimal example, suppose the state space is $\\mathbb{R}^2$, and we focus on the transition distribution from a particular $(s,a)$ pair where groundtruth is uniform over 4 points $(\\pm 1, \\pm 1)$. Loss of groundtruth model itself is $1+\\sqrt{2}/2$, which is higher than a wrong model that always predicts $(0,0)$ deterministically ($\\sqrt{2})$. \n\n---\n\n\n**Q1 \\& W3: Motivation for model-based selectors**\n\nWe understand the sentiment that the model-based selectors feel “extraneous”. In fact, we received similar comments in a previous round of review and decided to defer the contents to the appendix, only leaving a short summary in the main text. \n\nThat said, we argue that there is strong **theoretical** motivations for the model-based selectors. In particular, the selectors can leverage the information about Bellman operators (L207), which is accessible in the model-free setting, and obtain standard and elegant theoretical guarantees (Theorems 3 and 4) familiar to the RL theory literature. Our intuition is that **methods that properly leverage additional side information often performs better**, and the “standardness” and “elegance” of the guarantees are, in our opinion, enough motivation to consider the model-based selectors.\n\nIn fact, before we implement the model-based selectors, we were **fully anticipating** a trade-off between performance (that model-based should _outperform_ model-free) and applicability/computational cost (model-free applies to more settings and is cheaper to run). **The results come out as a big surprise, and we think reporting such negative results is invaluable to the research community.** This gets back to our comment on the research methodology: you said that the model-based methods lack motivation _because_ it performs poorly, which reflects the “empirical-first” methodology. In our theory-first approach, we develop the methods in theory and decide to include it **before we do any experiments**: if the experiments confirm our hypothesis, great; if not (as we saw), the surprisingly poor performance of model-based selectors further highlights the effectiveness of LSTD-tournament. As for why model-based selectors fail, we have some speculations (see response to ZupL’s (3)).\n\n---\n\n**Q3: Generalizability beyond Hopper and DDPG policies**\n\nThis is a valid point, and as usual, the more experiments on more environments/settings, the always better. That said, unlike most of RL research where training algorithms have many small design choices that can be tweaked to overfit to a setting, our loss is closed-form and optimization-free, leaving very little room for overfitting.\n\n---\n\n**Q1: Sec 2 pointer to Fig 5 describes Sec 5 protocol**\n\nFig 5 left describes the practical setup, which directly corresponds to the text in Sec 2."}, {"review_id": "zon7DUy2JS", "reviewer": "Reviewer_G4cF", "summary": "This paper focuses on the model selection for OPE itself. The authors develop: (1) new model-free and model-based selectors with theoretical guarantees, and (2) a new experimental protocol for empirically evaluating them. The authors exemplify the protocol on Gym-Hopper, and find that the proposed new model-free selector, LSTD-Tournament, demonstrates promising empirical performance.", "questions": "1. The authors assume the Q function follows a linear function approximation (line 141), which is not w.l.o.g. Are the following proofs based on this assumption?\n\n2. (optional suggestion) OPE and offline RL particularly benefit real-world applications where ground-truth evaluation is not available during the training and OPE process.  The authors would be better to show the effectiveness of the proposed methods in real-world experiments, such as robotics, e-commerce applications.", "limitations": "yes", "rating": 3, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strength:**\n1. The chicken-and-egg question that the paper focuses on sounds interesting. \n2. The paper improves the BVFT method and proposes two detailed methods. \n\n**Weakness:**\n\n1. Assumption Concern. The authors assume the Q function follows a linear function approximation (line 141), which is not w.l.o.g.\n\n2. Unclear Background. The basic idea of BVFT and the double sampling problem should be introduced in more detail and in an easier-to-understand way, e.g., in the Appendix. The readers may have no background knowledge of BVFT.", "was_revised": true, "final_justification": "As I am not familiar with the BVFT, my confidence in accepting or rejecting this paper is low. However, I think the paper writing needs to be improved. For a person who is not familiar with the related area, it is hard to follow.", "rebuttal": "We thank the reviewer for the comments. Unfortunately the reviewer **misunderstood a major technical point** of the paper regarding linear function approximation, which we explain below. We then respond to your other comments.\n\n---\n\n**We do NOT assume Q functions follow linear function approximation.**\n\nLine 141 is explaining the background knowledge of LSTDQ, which is indeed a method for learning Q-functions with linear function approximation (LFA). However, when we apply it to the model selection problem, **we do not make any assumptions on the Q-functions we select from**. They can be obtained by training neural networks using FQE on offline data (Figure 5 Left), or in our protocol, implicitly defined as the Q-function of the simulator MDP (Figure 5 Right). In any of these cases and in our theoretical development, we do not assume these candidate Q-functions to be generated/learned via LFA. The reason LFA is mentioned is because, when we select among the candidates, we use LSTDQ as a subroutine. LSTDQ indeed requires linear features, and these features are constructed from the candidate Q-functions themselves (Eq.6). Essentially we **reduce** the problem of selecting from **general candidate Q-functions** (which is difficult and very few solutions exist) to the subproblem of **learning with linear features** (which is well studied and has algorithms with strong guarantees), and how to do this reduction (without relying on additional hyperparameters or side knowledge) is the key innovation of our method. Also, if you skip the derivations, treat it as a blackbox, and jump right to the final loss (Eq.(10)), you will see that the equation is simple, closed-form, and has nothing to do with LFA. You asked whether the “proofs [are] based on this assumption” – as you can tell from the paper, no we don’t have this assumption in our main theoretical results (see Theorem 2 for example). \n\n---\n\n**Background of BVFT and double-sampling**\n\nBVFT is a complicated method and counterintuitive in many ways, and it takes very nontrivial efforts to grasp its main idea even for experts in offline RL theory if they have not seen the work before. So unfortunately we don’t think we can do justice to the method in the limited space. Our paper is written in a way that is meant to be self-contained, and theorists should be able to follow our derivations without needing to know BVFT in the first place, even if we did take the high-level spirit of BVFT (as acknowledged on Line 132) when designing our method. Also, we have provided theoretical and empirical comparisons to BVFT even if the readers don’t know the latter; for example, Line 53 and 173 mention that we have improved rates compared to BVFT, which can be appreciated without knowing the details of the method. \n\nFor double sampling, this is a well-documented issue first observed in 1995 [7] and also discussed in Sutton & Barto’s textbook [50] (see Chapter 11.6 in 2nd ed). For a concise explanation, see Section 3.1 of [9]. Again, for the purpose of our paper, the readers just need to know that TD-sq is biased, which we mentioned.\n\n---\n\n**Experiments in real-life applications**\n\nWe agree that this should be what the OPE community strives for. Unfortunately, 99% of the OPE works (in fact we don’t know *any* work that does it) don’t do that for understandable reasons: unlike most of ML where realistic experiments only require realistic data (e.g., ImageNet), RL experiments on real-life applications require engaging with the actual system. For OPE specifically, if we only have off-policy data, how do we know the groundtruth performance of the policy? The only way to get reliable groundtruth is to **run the target policy in the real-system**, and the difficulties in doing so is usually why OPE is needed in the first place. Even if technically possible, this requires closely working with domain experts and obtain access to real-life systems where decisions can cause financial and societal consequences (e.g., in online recommendation, you need to work closely with the company and get their permissions, which often requires an official affiliation with the company), which is a resource not many researchers have access to. All that said, we agree with the sentiment and believe the community as a whole should make efforts to get closer to real-life systems, though this is perhaps a luxurious request at the moment."}, {"review_id": "UGCl1Fs76D", "reviewer": "Reviewer_2FCh", "summary": "The paper studies the model selection problem in offline reinforcement learning. The paper studies both model-free and model-based selectors for off-policy evaluation (OPE) with theoretical guarantees and simulated experiments. For model-free selectors, the paper proposes a novel LSTD-Tournament, which exbits a smaller error rate compared to existing method, i.e., BVFT. For model-based methods, the authors studied regression-based selectors and sign-flip selectors.", "questions": "1. Maybe I'm unfamiliar with relevant literature, but can you obtain lower bounds for model-free and model-based error rates? How tight are your proposed methods? \n\n2. In figure 2, can you explain why mb_naive performs well in the top row but has disastorous error in the last two in bottom row?", "limitations": "The authors addressed the limitations.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strenths:\n1. The paper studies an interesting model selection problem for OPE instances. Both types of methods have nice theoretical guarantees compared to baselines.\n\n2. The experiments are also extensive and support the theoretical claims.\n\nWeaknesses:\n\n1. As the authors also stated, the assumption that Q-function candidates may be unrealistic in practice.\n\n2. In Section 5, even though the theorems are deferred to appendix, I suggest the authors to summarize them briefly in the main text.", "was_revised": true, "final_justification": "See the discussion with authors.", "rebuttal": "We thank the reviewer for the positive evaluation, and respond to your questions below.\n\n---\n\n**Maybe I'm unfamiliar with relevant literature, but can you obtain lower bounds for model-free and model-based error rates? How tight are your proposed methods?**\n\nIf by rates you mean the dependence on n, then $n^{-1/2}$ is the best you can get for simple reasons: OPE is clearly more general than estimating the mean of a bounded random variable and subsumes the latter as a special case. For mean estimation, without additional structures and assumptions, the $n^{-1/2}$ rate is the minimax rate and well-established in the statistics literature, so it is simply impossible to do better than that in general. That said, our bounds not only depend on n but also crucially depend on the coverage parameters. The problem is that different methods depend on different forms of coverage parameters  (e.g., the $\\sigma\\_{\\min}$ term in Theorem 2, $C^\\pi$ in Theorem 4, and $C\\_\\infty^\\pi$ in Theorem 5), which are often not comparable. We have a nuanced discussion on this topic in Appendix C.5 for RL theorists readers familiar with the literature. \n\n---\n\n**In figure 2, can you explain why mb\\_naive performs well in the top row but has disastrous error in the last two in bottom row?**\n\nAs mentioned on Line 203, this is because mb\\_naive strongly biases towards deterministic models when the candidate models have different levels of stochasticity. In the top row of figure 2 all models have similar levels of stochasticity, so this problem is not exposed. In the bottom row, however, columns 2 and 3 correspond to cases where the groundtruth model has relatively high stochasticity, so the bias is fully exposed. In contrast, in column 1, the groundtruth model is the most deterministic one in the family, so mb\\_naive’s bias acts in its favor.\n\nYour question (Reviewer Robe also asked the same) made us realize that the bias of mb\\_naive is much less known than we thought, so here is a more detailed explanation of the underlying math (which we can add to the appendix): The key is that in the loss we have $\\\\\\| s’ - \\tilde{s}\\\\\\|$ (Line 120), where $s’$ is from the data and $\\tilde{s}$ from the candidate model. When the candidate is the true model, $s’$ and $\\tilde{s}$ are iid, but **their randomnesses are independent and “out of sync”**, which causes the loss to be non-zero and scale with the inherent randomness of the distribution. As a minimal example, suppose the state space is $\\mathbb{R}^2$, and we focus on the transition distribution from a particular $(s,a)$ pair where groundtruth is uniform over 4 points $(\\pm 1, \\pm 1)$. If we calculate the expected loss of groundtruth model itself (on data generated from the same model), we get $1+\\sqrt{2}/2$. However, a wrong model that always predicts $(0,0)$ deterministically will get a lower loss of $\\sqrt{2}$. \n\n\n---\n\n**Authors should summarize the theorems briefly in the main text.**\n\nWe will surely try to!"}, {"review_id": "BRabyKYrwJ", "reviewer": "Reviewer_ZupL", "summary": "This paper addresses a fundamental challenge in offline RL that inhibits its practical applicability: How does one select the best off-policy evlauation (OPE) method? If a trusted simulator of the environment is not available, it is challenging to select a candidate of methods to evaluate a set of trained policies. The authors contribute bits to a multitude of topics including the introduction of selection algorithms (LSTD-Tournament, model-based selectors), theoretical analysis, an experiment protocol and empirical validation on the Gym-Hopper environment.", "questions": "(1) Can you restructure the paper in a way that Figure 5 is included in the paper instead of the appendix? To me, Figure 5 is a vital part to understand the paper. (could help B, C, D)\n\n(2) Is \"model selection for OPE\" an established term? If so, can you provide references to this, and if not, can you elaborate why the term is appropriate, especially with the multitude of meanings of the phrase \"model\" in the OPE context? (B, D)\n\n(3) Can you elaborate more in-depth your observations of model-based selectors? It is surprising to me that they do not hold your expectations. (F)\n\n(4) Can you improve the clarity in the abstract and in the introduction (especially the first paragraph) to better reflect the topics of the paper? (B, D)\n\n(5) Can you include a small conclusion, maybe would fit the start of section 7, that wraps up the paper? (D, E)\n\n\n\nI will reconsider my \"Rating\" if these questions are addressed.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Major Weaknesses:\n\n(A) The combination/selections of contributions is to my understanding rather unconventional. (Quality-related)\n\n(B) The phrasing of \"what the paper is about\" lacks clarity, and in my opinion is prone to be misunderstood. (Clarity-related)\n\n(C) An essential part, namely Figure 5, that is vital to the comprehensibility of the paper is pushed into the appendix. (Quality/Clarity-related)\n\nMinor Weaknesses:\n\n(D) Due to the high-level perspective, the topic is generally hard to grasp. \n\n(E) The paper lacks a concise conclusion which wraps up the multitude of topics. It has a thorough exemplification of its protocol in 6 and limitations/future work in 7. \n\n(F) The discussion of the results in 6 could be clearer.\n\nStrengths:\n\n(G) The paper addresses an important and understudied gap in the literature.\n\n(H) The paper presents significant insights, generally the empirical resutls of model-based and model-free methods is very appreciated and especially the presented subgrid studies are interesting to me.\n\n\nPlease address the points under \"Questions\".", "was_revised": true, "final_justification": "Considering the rebuttal and the various discussions between authors and the other reviewers, I increased my score to **4: Borderline accept**. \n\nThe paper covers ideas in a promising direction, highlighting a gap in the current literature that is under-researched despite its important practical implications. While I'm inclined to be more positive about the paper, I totally agree with the critique that the paper's presentation leaves room for improvement. With the overall organisation, selection of topics and a rather vague central thread in mind, the paper seems difficult to digest.\n\nOverall, I'm still in favour of this submission's acceptance.", "rebuttal": "We thank the reviewer for the helpful comments and feedback. We note that you rated “3: good” in all subcategories but gave “borderline reject” in overall rating. We hope you can reconsider the rating after seeing our response and are glad that you are open to it (“I will reconsider…”).\n\n---\n\n**(A) The combination/selections of contributions is to my understanding rather unconventional. (Quality-related)**\n\nWe agree that the combination of contributions is unconventional (theory + experiment protocol + empirical), but respectfully disagree that this hurts the quality of the paper. These contributions emerge naturally when we investigate the topic: after developing the methods and their theories, we want to empirically test them, but the protocols in the prior works have been hard to use and caused many troubles in our replication (see e.g., Footnote 1 on page 2). So we come up with a new protocol that is much better in producing stable candidates and enable interesting studies (Sections 6.2 and 6.3) that aren’t possible before. \n\n---\n\n**(1) Restructure the paper to include Figure 5 in main text**\n\nThanks for the great suggestion. We totally agree that Figure 5 can be very helpful in understanding the overall problem and protocols, especially given that the complications in offline evaluation and selection are not something that most of the RL researchers have thought about carefully. We had to defer the figure to the appendix due to space limit, but if the paper is accepted, we fully intend to include it in the main text given the 1 extra content page. \n\n---\n\n**(2) Is “Model selection in OPE” an established term? especially with the multitude of meanings of the phrase \"model\"**\n\nWe agree that the word “model” is heavily overloaded in OPE and RL in general. As we mentioned, model selection in OPE is a heavily under-investigated problem, so it’s hard to say it’s “well-established” given the scarcity of research in this area in the first place, but prior work like [62] did also use “offline model selection”. In addition, despite the multitude of meanings of “model”, “model selection” as a whole is a well-established technical term in statistics, and we are following this convention. In machine learning, model selection is closely related to concepts like cross validation and hyperparameter tuning (which we mention in the first sentence of abstract to help resolve the ambiguity of “model selection”), but the latter two refer to something more specific (e.g., cross validation refers to procedures like k-fold, and hyperparameter tuning often applies to settings where candidate functions/predictors are generated by the same algorithm with different hyperparameters) that do not necessarily apply to our problem setting, and the term “model selection” is the more general and abstract term which we think fit better.\n\n---\n\n**(3) surprising to me that [model-based predictors] do not hold your expectations.**\n\nIndeed! Our rationale was that model-based predictors enjoy strong theoretical motivations: they apply to narrower settings (can only select from models, not Q-functions; Line 105), but leverage more information from the model (the Bellman operators); in particular, not having access to Bellman operators is _the_ biggest difficulty in model-free selection (Line 207), but the operators can be learned quite straightforwardly in the model-based setting (Line 208). **As a general wisdom, methods that properly leverage additional side information often perform better.** Moreover, model-based selectors enjoy more standard and interpretable guarantees found in the offline RL theory literature (Line 360). So our anticipation was that model-based predictors would outperform model-free ones, which is a trade-off between performance (model-based wins) and applicability (model-free wins). Therefore, the fact that model-based performs worse is definitely surprising, and **we view this a great opportunity to report interesting _negative_ results, which we hope is more appreciated and encouraged by the community**, and the surprisingly poor performance of model-based selectors further highlights the effectiveness of LSTD-tournament. \n\nAs for why model-based selectors fail, we have had some preliminary investigation into this, which we can add to the appendix; one possible reason is that these estimators are pretty sensitive to small errors (e.g., the Monte-Carlo estimates still has small nontrivial errors despite we use $l=128$ rollouts (Line 288) which is quite computationally intensive (Appendix D.3)), and the model-based estimator suffers more due to its multi-stage process. In contrast, LSTD-tournament has an interesting property that it is largely immune to the Monte-Carlo errors (Appendix E.3), since the noise in each Q-estimate will be averaged out over the entire dataset due to the lack of absolute value or square inside the loss function (Eq.(10)).\n\n---\n\n**(4) Can you improve the clarity in the abstract and in the introduction (especially the first paragraph) to better reflect the topics of the paper? (B, D)**\n\nWe are definitely happy to but it is unclear which part causes confusion from your review. In particular, your summary of the paper seems spot on and we don’t see any obvious misunderstanding in your text. If you could elaborate more on where it needs clarification, we can improve the clarity of the text accordingly.\n\n---\n\n**(5) Can you include a small conclusion, maybe would fit the start of section 7, that wraps up the paper? (D, E)**\n\nDefinitely happy to and we plan to do so if the paper is accepted and given one extra content page."}]

Author rebuttal:

Meta-review: This paper addresses the problem of hyper-parameter selection in off-policy evaluation (OPE) which is of critical performance for the safe deployment of RL agents and an important problem for the RL community. Many OPE methods have hyper-parameters that cannot be practically tuned; this motivates the need for hyper-parameter free methods that can identify which of a set of OPE estimates is going to produce the most accurate estimate. This paper is primarily theoretical in nature. It develops two methods: one model-free and one model-based for OPE estimator selection that show improved convergence rates. The paper also claims a new experimental protocal for evaluating the quality of learned value functions and dynamics models and demonstrates this protocal on the Hopper gym environment. The empirical evaluation shows that the new method provides more reliable value function assessments than the prior BVFT method that could be considered the prior leading method for this problem.

The paper makes an important contribution to the theoretical foundations of OPE and addresses a problem of great practical value. Theoretical results go beyond prior work and are correct to the best of the reviewers knowledge. The work seems to overcome limitations of the prior BVFT method and has improved theoretical guarantees.

A couple of concerns were discussed by the reviewers concerning the claims made and presentation of the paper. These are (1) disagreement over the extent to which the method is truly hyper-parameter free and (2) the model-based section of the paper seemed a little disconnected from the rest of the paper. I will add to these concerns that the paper claims a new experimental protocal but it seems very similar to the work "High-confidence error estimates for learned value functions" by Sajed et al. (2018).

Overall, the paper makes a solid theoretical contribution and shows improved empirical performance on an important problem and the reasons to accept are stronger than the limitations raised. All concerns could be addressed in the camera-ready version.

Suggestions to authors:
1. Discuss alternative design decisions regarding the feature map even if these are left to future work.
2. Add more motivation for the model-based estimators.
3. Either tone down the claim about a new experimental protocal or discuss how it differs from prior work. I don't see this as a major contribution of the paper but of course if there are novel elements then please highlight them.
Decision: Accept (poster)