Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper presents an algorithm for model selection with nested model classes when used with value function approximation for offline RL. The bound achieved by the proposed algorithm is minimax rate-optimal oracle inequalities (upto logarithmic factors). 
strengths_and_weaknesses: Strengths:
- The problem and the related algorithm are new to my knowledge
- The paper is well written
- The theory is accompanied by some reasonable experimental support.

Weaknesses:
- The main issue I have with this paper is how do we obtain a set of nested function classes where such an ordering is actually satisfied? Clearly, the authors recognize this issue in footnote 3, and this, in a sense breaks much of what is developed in the paper since the paper posits the existence of nested model classes that satisfy the approximation error-statistical complexity trade-off. These issues are pretty much mirrored in existing works, where there is a need to select a model class for performing estimation and it is unclear how to tradeoff approximation and estimation error. I would recommend the authors discuss this in greater detail.
- Another issue i see is while the authors stress schemes that achieve rate-optimality, the dependence on the approximation error appear to be sub-optimal, and this is the dominating term (particularly in the limit).
- The experimental results are limited at best. I would recommend that these be expanded in scope to consider continuous control tasks (e.g. with D4RL benchmark suite) since many methods used for these tasks rely on variants of actor-critic methods. I do like the paper's result, although i am trying to understand if the authors could compensate for what appears to be non-trivial limitations in the problem setup through more extensive empirical studies.
questions: Pretty much all my questions are summarized in the weaknesses above.
limitations: I'd recommend the authors consider any unintended implications of their paper's result, although, I do not see any specific limitations myself.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper proposes an algorithm for offline reinforcement learning.
strengths_and_weaknesses: Strength: This paper provide an algorithm and upper bound its suboptimality.

Weakness. The relation and difference of their work and other works in offline RL is unmentioned.
questions: What is the relation and difference of this paper and other works for offline RL, including [1], [2], and [3].

[1]. Xie, Tengyang, et al. "Bellman-consistent pessimism for offline reinforcement learning." Advances in neural information processing systems 34 (2021): 6683-6694.

[2]. Zhan, Wenhao, et al. "Offline reinforcement learning with realizability and single-policy concentrability." Conference on Learning Theory. PMLR, 2022.

[3]. Uehara, Masatoshi, and Wen Sun. "Pessimistic model-based offline reinforcement learning under partial coverage." arXiv preprint arXiv:2107.06226 (2021).

limitations: The relation and difference of their work and other works in offline RL is unmentioned.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper considers the model selection problem for offline RL where the learner is given a nested set of model classes and must choose which one to use completely offline without interacting with the environment. The paper extends previous work from the contextual bandit setting to the RL setting and achieves minimax optimal rates up to log factors for a novel algorithm ModBE. Importantly, ModBE relies on a one-sided generalization test where regressors from two models classes are compared on the same regression problem induced by the Bellman targets of the smaller model class in the nested ordering. Some empirical results are presented on small domains.
strengths_and_weaknesses: Strengths:

1. The technique of applying the one-sided test on pairs of model classes for the same Bellman backup regression problem is clever and novel. The novel technique could be useful to the community as it is possible that the core idea could find uses in other problems around RL.

2. The proofs seem to be correct, although I haven't checked everything in the appendix carefully.

Weaknesses:

1. In general it seems that one should expect C(\mu) to be exponential in H. The lower bound from LTND21 says that dependence on a comparator policy is not possible, so the authors argue that this term is fundamental. But at that point, the main results of the paper seem to really only apply to short horizon problems. For any longish horizon, this constant will likely be enormous and the optimal rates may not really matter. It is still nice to have the optimal rates in n, but this seems like a serious issue that is sort of glossed over in the paper. It is possible that the problem as stated really is just too hard and that we should instead be considering more structured settings where the exponential lower bound could be avoided rather than proving bounds that are exponential.

2. The experiments do not seem to show much benefit from the ModBE algorithm over the naive hold-out baseline. On the bandit and cartpole problems there is no noticeable difference at all. On mountain car ModBE seems slightly better, but never enough that the errorbars do not overlap with the baseline (and it seems that performance in the dataset is not good enough to learn a decent policy, no matter the model class). In my mind, these experiments cannot be seen as evidence of the usefulness of ModBE over the much simpler (both conceptually and implentation-wise) baseline.

3. In practice, the ModBE algorithm itself has a hyperparameter in the form of the tolerance value. In the theory this can be set to some well-motivated value, but in practice it is often the case that those values will be overly conservative (as in most RL theory to practice conversions). For example, in the experiments the authors use a heuristic to set the tolerance based on pseudodimension bounds, but this may not be accessible with more realistic (i.e. larger neural net) model classes and then the parameter would need to be tuned. There is no discussion about sensistivity to this choice in practice and it seems like a serious issue for an algorithm that attempts to do model selection to potentially be hyperparameter sensitive itself.

Summary:

Overall, I think the paper studies an important problem and presents a clever algorithm and proof, but am wary of the significance of the contribution on a few levels: it seems that the assumptions effectively render the bound exponential in most realistic problems and from a practical perspective the paper does not provide clear evidence on the usefulness or robustness of the method.
questions: Is it clear that we need to make assumption 1 or just that a simple assumption with one comparator policy is not sufficient? If we need assumption 1, how important are optimal rates if dependence on H is exponential regardless? Should we consider a more structured set of problems to avoid these issues and yield more practical insights?

Can you provide empirical evidence where ModBE is clearly beneficial? Or even a toy problem where hold-out fails but ModBE succeeds?
limitations: The paper is sufficiently abstract that I think it is ok that it does not explicitly address ethical concerns in the main text.
review_title: null

Meta-review: metareview: This paper studies the model selection problem in the offline RL setting. The paper focuses on a theoretical study where a sequence of nested models are provided and the algorithm ought to output a class that nearly matches the optimal one (the so-called oracle inequality). It is surprising that the proposed algorithm is simple (QVI + generalization testing) but can achieve the oracle inequality. Both the meta-reviewer and some of the reviewers believe that the paper have a solid theoretical contribution and is qualified for publication in NeurIPS.

However, in addition to the issues mentioned by the reviewers, the meta-reviewer finds that the presentation could be additionally improved. For instance, in the current form of the paper, it is hard to understand the general idea by just reading the first 8 pages -- e.g., one might simply think to estimate each model separately and then choose the best one; why this simpler algorithm does not work? It would be beneficial if the authors could provide a technical overview in the main text rather than provide it in the appendix. Other minor issues including the notations. In Def. 1, the meaning of w is not provided; w was first mentioned, then then in the equations what was used is w_{n, \delta}; also, it is said w is a function -- but it is a function of what? n, delta, and F_k?

With that being said, the recommendation of this paper is an accept. The meta-reviewer encourages the authors to incorporate the reviewers' comments and further improve the presentation.
confidence: Certain
Decision: Accept