Title: Model Selection for Off-policy Evaluation: New Algorithms and Experimental Protocol
Venue: NeurIPS 2025
Accepted: yes
Keywords: model selection

off-policy evaluation
Abstract: Holdout validation and hyperparameter tuning from data is a long-standing problem in offline reinforcement learning (RL). A standard framework is to use off-policy evaluation (OPE) methods to evaluate and select the policies, but OPE either incurs exponential variance (e.g., importance sampling) or has hyperparameters on their own (e.g., FQE and model-based). We focus on hyperparameter tuning for OPE itself, which is even more under-investigated. Concretely, we select among candidate value functions ("model-free") or dynamics models ("model-based") to best assess the performance of a target policy. We develop: (1) new model-free and model-based selectors with theoretical guarantees, and (2) a new experimental protocol for empirically evaluating them. Compared to the model-free protocol in prior works, our new protocol allows for more stable generation and better control of candidate value functions in an optimization-free manner, and evaluation of model-free and model-based methods alike. We exemplify the protocol on Gym-Hopper, and find that our new model-free selector, LSTD-Tournament, demonstrates promising empirical performance.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses a fundamental challenge in offline RL that inhibits its practical applicability: How does one select the best off-policy evlauation (OPE) method? If a trusted simulator of the environment is not available, it is challenging to select a candidate of methods to evaluate a set of trained policies. The authors contribute bits to a multitude of topics including the introduction of selection algorithms (LSTD-Tournament, model-based selectors), theoretical analysis, an experiment protocol and empirical validation on the Gym-Hopper environment.\n\nSTRENGTHS AND WEAKNESSES:\nMajor Weaknesses:\n\n(A) The combination/selections of contributions is to my understanding rather unconventional. (Quality-related)\n\n(B) The phrasing of \"what the paper is about\" lacks clarity, and in my opinion is prone to be misunderstood. (Clarity-related)\n\n(C) An essential part, namely Figure 5, that is vital to the comprehensibility of the paper is pushed into the appendix. (Quality/Clarity-related)\n\nMinor Weaknesses:\n\n(D) Due to the high-level perspective, the topic is generally hard to grasp. \n\n(E) The paper lacks a concise conclusion which wraps up the multitude of topics. It has a thorough exemplification of its protocol in 6 and limitations/future work in 7. \n\n(F) The discussion of the results in 6 could be clearer.\n\nStrengths:\n\n(G) The paper addresses an important and understudied gap in the literature.\n\n(H) The paper presents significant insights, generally the empirical resutls of model-based and model-free methods is very appreciated and especially the presented subgrid studies are interesting to me.\n\n\nPlease address the points under \"Questions\".\n\nQUESTIONS:\n(1) Can you restructure the paper in a way that Figure 5 is included in the paper instead of the appendix? To me, Figure 5 is a vital part to understand the paper. (could help B, C, D)\n\n(2) Is \"model selection for OPE\" an established term? If so, can you provide references to this, and if not, can you elaborate why the term is appropriate, especially with the multitude of meanings of the phrase \"model\" in the OPE context? (B, D)\n\n(3) Can you elaborate more in-depth your observations of model-based selectors? It is surprising to me that they do not hold your expectations. (F)\n\n(4) Can you improve the clarity in the abstract and in the introduction (especially the first paragraph) to better reflect the topics of the paper? (B, D)\n\n(5) Can you include a small conclusion, maybe would fit the start of section 7, that wraps up the paper? (D, E)\n\n\n\nI will reconsider my \"Rating\" if these questions are addressed.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

comments: SUMMARY:
The paper studies the model selection problem in offline reinforcement learning. The paper studies both model-free and model-based selectors for off-policy evaluation (OPE) with theoretical guarantees and simulated experiments. For model-free selectors, the paper proposes a novel LSTD-Tournament, which exbits a smaller error rate compared to existing method, i.e., BVFT. For model-based methods, the authors studied regression-based selectors and sign-flip selectors.

STRENGTHS AND WEAKNESSES:
Strenths:
1. The paper studies an interesting model selection problem for OPE instances. Both types of methods have nice theoretical guarantees compared to baselines.

2. The experiments are also extensive and support the theoretical claims.

Weaknesses:

1. As the authors also stated, the assumption that Q-function candidates may be unrealistic in practice.

2. In Section 5, even though the theorems are deferred to appendix, I suggest the authors to summarize them briefly in the main text.

QUESTIONS:
1. Maybe I'm unfamiliar with relevant literature, but can you obtain lower bounds for model-free and model-based error rates? How tight are your proposed methods? 

2. In figure 2, can you explain why mb_naive performs well in the top row but has disastorous error in the last two in bottom row?

LIMITATIONS:
The authors addressed the limitations.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

comments: SUMMARY:
This paper focuses on the model selection for OPE itself. The authors develop: (1) new model-free and model-based selectors with theoretical guarantees, and (2) a new experimental protocol for empirically evaluating them. The authors exemplify the protocol on Gym-Hopper, and find that the proposed new model-free selector, LSTD-Tournament, demonstrates promising empirical performance.

STRENGTHS AND WEAKNESSES:
**Strength:**
1. The chicken-and-egg question that the paper focuses on sounds interesting. 
2. The paper improves the BVFT method and proposes two detailed methods. 

**Weakness:**

1. Assumption Concern. The authors assume the Q function follows a linear function approximation (line 141), which is not w.l.o.g.

2. Unclear Background. The basic idea of BVFT and the double sampling problem should be introduced in more detail and in an easier-to-understand way, e.g., in the Appendix. The readers may have no background knowledge of BVFT.

QUESTIONS:
1. The authors assume the Q function follows a linear function approximation (line 141), which is not w.l.o.g. Are the following proofs based on this assumption?

2. (optional suggestion) OPE and offline RL particularly benefit real-world applications where ground-truth evaluation is not available during the training and OPE process.  The authors would be better to show the effectiveness of the proposed methods in real-world experiments, such as robotics, e-commerce applications.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper attempts to address the problem of model selection for off-policy evaluation (OPE) algorithms. It introduces a hyperparameter-free model-free selector named LSTD-Tournament, derived from the LSTDQ algorithm, and provides an accompanying theoretical analysis. The paper also proposes several new model-based selectors and a new experimental protocol for evaluating OPE selectors. This protocol generates candidate Q-functions by perturbing the parameters of a ground-truth simulator. Experiments on the Gym-Hopper environment are presented to evaluate the proposed methods.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- **Relevant Problem Formulation**: The paper focuses on a significant and challenging problem in practical offline RL: how to select among different OPE estimators or their configurations. This is a critical step for reliable policy selection before deployment.\n- **Solid Theoretical Guarantees**: The LSTD-Tournament selector is supported by a clear theoretical analysis (Theorem 2), demonstrating a standard $n^{−1/2}$ convergence rate, a notable improvement over BVFT.\n- **An Attempt at a Better Protocol**: The proposed experimental protocol tries to solve a real problem with existing benchmarks that rely on unstable FQE instances. The idea of generating candidates from perturbed ground-truth models is an interesting direction for creating more controllable testbeds.\n## Weaknesses\n1. **Clarity and Organization**: One of the paper's primary weakness is its poor organization and clarity, which makes it extremely difficult to follow. The flow of information is chaotic and confusing.\n    - *Disruptive Forward References*: The text is littered with forward references that disrupt the reading flow and require the reader to jump to later parts of the paper (or the appendix) before the necessary context is provided. For instance:\n        - In the Preliminaries (Section 2), the text directs the reader to \"see Figure 5 for a visualization\", but Figure 5 is in the appendix and illustrates the experimental protocol from Section 5.\n        - Section 3.1 refers to \"e.g., \\mathcal{G} in Section 4\", a concept not yet explained.\n        - Section 5.2 (Computational Efficiency) tells the reader to \"see Section 6.2\" for an example of its application, jumping ahead of the entire experimental setup.\n    - *Vague Justification of Claims*: Some claims are not adequately explained. For example, when discussing the naive model-based baseline, the paper claims there is \"empirical evidence of this in Section 6 (see `mb_naive` in Figures 2 and 3)\" but provides no further interpretation. Looking at the plots, it is not immediately obvious how they serve as damning evidence; in several high-noise environments (e.g., top row of Figure 2), `mb_naive` appears to perform quite well or at least comparably to other methods.\n    - *Minor Typos*: There are some typos, such as \"textitunits\" in Section 5.1.\n2. **Misleading \"Hyperparameter-Free\" Claim and Unmotivated Design**: The paper's central claim of LSTD-Tournament being \"hyperparameter-free\" is questionable. The design of the feature map $\\phi(s,a)$ (e.g., $[Q_i, Q_j]^T$ or the normalized difference version) is a critical design choice with a significant impact on performance, effectively acting as a hyperparameter. In fact, the weighting function $Q_k(s,a)$ in Eq.10 can be any function of $(s,a)$, effectively introducing infinitely many degrees of freedom. The paper acknowledges variants (Section 3.2, Appendix E.2) but heuristically fixes one for the main experiments, which merely hides this degree of freedom rather than eliminating it. The ablation study in Appendix E.2 demonstrates that the performance of LSTD-Tournament is quite sensitive to choice of the weighting function. This core design choice lacks a strong intuitive motivation and is not properly justified through ablation studies comparing it with simpler alternatives, e.g., random functions of $(s,a)$.\n2. **Underdeveloped Model-Based Contribution**: The section on new model-based selectors (Section 4) feels extraneous and not well-motivated.\n    - *Poor Performance and High Cost*: These selectors are empirically shown to be outperformed by the simpler, more efficient, and more broadly applicable model-free LSTD-Tournament. They also have a higher computational cost ($O(m^2)$), making them less practical.\n    - *Lack of Motivation*: The paper offers little intuition for the design of these new selectors (especially the \"Sign-flip Selector\"). It is unclear what benefits they were intended to provide over existing approaches or the model-free methods.\n3. **Limited Diversity in Experiments**: The generalizability of the empirical findings is constrained by a lack of diversity in the experimental setup.\n    - *Single Environment*: All experiments are conducted exclusively on the Hopper-v4 environment. Although the authors introduce variations by modifying parameters like gravity and noise, the underlying task structure remain the same. Testing on a wider range of distinct environments would significantly bolster the claims of robustness.\n    - *Homogeneous Policies and Datasets*: The target policies are all generated from checkpoints of a single algorithm, DDPG. The behavior policies are also created in a uniform way, typically by adding Gaussian noise to a DDPG policy. A more realistic scenario would involve selecting between policies generated by different types of algorithms (e.g., CQL, IQL, TD3) and using offline datasets collected from varied sources, which this study does not cover.\n\nQUESTIONS:\n1. Could you clarify the motivation behind the design of the new model-based selectors? What advantages were they hypothesized to have?\n2. Regarding the claim that the naive model-based selector fails in stochastic environments, could you elaborate on how Figures 2 and 3 demonstrate this? A more direct explanation in the text would be very helpful, as the visual evidence alone is not entirely clear.\n3. Could you comment on the generalizability of your results beyond the Hopper environment and DDPG-based policies?\n4. The LSTD-Tournament loss uses candidate functions $Q_k$ or the normalized difference to weigh the Bellman residual of $Q_i$. Could you provide a more concrete intuition for why this is a good design choice? Have you considered or experimented with other weighting functions to verify that using the candidate Q-functions themselves is critical to the method's success?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}