Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper focuses on a hyperparameter-free policy selection for offline reinforcement learning. The authors suggests two methods: (1) a automatic value-function selection method for discrete action spaces using Batch Value Function Tournament (BVFT), (2) a combination of OPE algorithms and BVFT variants for continuous action spaces (BVFT-PE, BVFT-PE-Q, ...)
main_review: Offline policy selection is an important problem for deploying offline reinforcement learning algorithms to many practical real-world problems. However, previous methods using off-policy policy evaluation (e.g. FQE) also require hyperparameter tuning, which makes a chicken-and-egg situation. This paper tackles this issue in a principled way using BVFT, which is the value function selection method with theoretically advanced Bellman error estimates. This paper has strong motivations and the methods are appropriate. Hence, I vote for the acceptance to this paper. I have following questions for this paper:

1. In line 267, BVFT-PE-Q loss uses a hyperparameter $\lambda$ which adjusts a penalization coefficient of the average critic value. (maybe hence the authors call their methods as "nearly"-hyperparameter-free in Section 6) How is $\lambda$ choiced for the experiment and how sensitive? 

2. How many FQE instances are used for multi-FQE experiments (Figure 5L)? (maybe described as $L$ in the paper). I think it will be good to show that a choice of L does not significantly affect the result of experiment.



limitations_and_societal_impact: Yes, the authors adequately addressed the potential negative societal impact of their work.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: Batch Value Function Tournament (BVFT) is an existing approach to offline policy selection that uses piecewise linear value function approximations and selects the policy with the smallest projected Bellmane error in that space. This paper automatically selects the BVFT value function resolution by using a clever argument involving the convergence of the projected Bellman error to the regular Bellman error when the resolution goes to zero. On datasets from Atari and Gym datasets this approach is very close to selecting the best resolution in retrospect. They also show how this approach can be extended to policy gradient methods for continuous actions, where the application of BVFT is less straightforward than in discrete action spaces.

main_review: To me the paper ranks quite highly in terms of originality - it can be seen as a relatively simple extension of an existing approach, but as I mentioned in the summary the extension is quite clever and seems to work really well on some problems and is well motivated. I also found the  paper very good in terms of quality and clarity

In terms of significance, I think the authors should explore more what kind of domains their approach can be really helpful in. My feeling is that it will be less useful in domains where complex function approximators are required for learning good policies, and their approach to continuous action space policy selection seems a bit complex for widespread practical use to me, plus it introduces lambda which is a hyperparameter that probably needs tuning (although the approach is interesting). I do think however that there is at least one important application area that looks perfectly suited for their approach, and that is OPE / policy selection from clinical trial data - see e.g. Raghu, A., Komorowski, M., Celi, L. A., Szolovits, P., and Ghassemi, M. (2017) Continuous state-space models for optimal sepsis treatment: a deep reinforcement learning approach. This is because the data there is limited but piecewise continuous value functions are likely to work well there, and also be more interpretable than neural approximations. I encourage the authors to explore this direction if they find it interesting.

limitations_and_societal_impact: I appreciate the authors pointing out one of the major limitations of their work - “the success of BVFT relies on the existence of a reasonable approximation of Q* among the candidate functions”, and the fact that they explore (and to some extent address) the limitations in the context of continuous action spaces. And while I understand why this is not included in the paper, I definitely encourage the authors to try and produce results using the benchmarks in [Fu+21].

As mentioned in the main review, I think using this type of approach with clinical trial data for adaptive treatment regimes could have considerable positive societal impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposed a new method Batch Value-Function Tournament (BVFT) for the off-policy evaluation of RL. The algorithm is hyperparameter-free thus could be applied to a variety of Offline RL algorithms. Experiments are conducted on both discrete and continuous control benchmarks. BVFT outperforms other baselines on some Atari (discrete control) environments but cannot match OPE methods on the Mujoco (continuous control) environments. The authors further consider a method combining BVFT and OPE. This method is able to tie with the performance OPE on Mujoco and has some nice properties. In addition to empirical results, the authors also provide theories to support their method.

main_review: Off-policy evaluation of RL is indeed an important open question for Offline RL. The authors address it with a novel method, with extensive comparison to the existing OPE methods and some other natural approaches based on quantities like Bellman Residual, Average Q value, etc.

The clarity of this paper is ok but could be improved. I appreciate the thoughtful writing with little typos. The authors often try to ask questions then answer them. But sometimes the answer is not supported with convincing evidence.

Moreover, the authors provide some theoretical properties of their BVFT algorithm. In particular, Proposition 2 which proves the equivalence between BVFT-loss and Bellman residual is interesting. However, it is not very clear how this relates to the BVFT in implementation (I see the authors provide some discussion, but giving more augmentation or insight would enhance the flow of logic).

Here are some confusions/suggestions:
On the experiment part, using the 2 quantities top-k regret and top-k precision as the evaluation score is straightforward and considerate. But it would be more clear if the authors provide a discussion on their (different) meaning alongside the definition. (I would have some explanations on these quantities but I am not sure if the authors have the same thought.) 
To study the variance, the authors also consider the randomness introduced by datasets. It would also be good to consider randomness introduced by the algorithm training process (initialization and SGD).
BCQ and CQL are both experimented on the Atari benchmark, but I wonder why the authors just consider BCQ on Mujoco? These 2 offline RL algorithms are substantially different and it would be interesting to see comprehensive results of both.
Some of the algorithmic discussion in the appendix could be moved to the main paper, while some derivations in Sections 3 and 4 could be moved to the appendix, if they are not essential for understanding the main idea of the algorithm. It would also be good if the authors provide more discussion on the intuition of their approach.

*post-rebuttal*:
I thank the authors for their detailed reply. I think it would be good for to include

I think this paper studies an important question and is illuminating in some sense. I consider raising the score if the above are addressed.

*

limitations_and_societal_impact: In section 6 the authors have addressed most of the limitations of their work. One additional limitation is that their method would mostly be restricted to policy selection for Q-learning based algorithms. Though this includes a large family of offline RL approaches, imitation learning based, or model based algorithms also achieve promising results in the offline RL problem, and are worth studying. The authors could try to discuss how to extend BVFT on addressing these algorithms.

The BVFT algorithm differs from previous OPE methods in the idea and implementation, and achieves SOTA performance on discrete control benchmarks. But to avoid performance degradation in continuous control, the authors combine BVFT and OPE. Despite some advantages over OPE, this introduces additional complexity into the implementation and makes it unclear which part contributes to a promising policy selection. As for offline RL algorithms, the need for distributional constraint or value pessimism is concentrated on continuous control tasks. The situation is more interesting here, and the paper could have provided more insight. Nevertheless, it would be more significant if there is a unified approach that performs consistently well across discrete and continuous control benchmarks.

I don’t see any potential negative societal impact of this work.
review_title: null

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: Offline Policy Evaluation (OPE) is when one needs to choose the best policy within a set of policies given only access to an offline dataset. Most OPE algorithms relies on function approximation and thus has hyperparameters. Previous works showed promising performance when the hyperparameters are tuned “online”, thus limiting their use case in practice. This paper proposes a way to select hyperparameters using purely the given offline dataset. 
 
This paper expands on BVFT and turns it into a practical OPE algorithm without hyperparameters. More specifically, it proved that the BVFT-loss in its limit is equivalent to the naive estimator of the Bellman residual (which is biased in a stochastic environment). When one chooses a reasonable discretization resolution, experimentally BVFT performs better than the naive estimator. When the environment is continuous, the authors proposed to combine BVFT with FQE or other hyperparameter-dependent OPE methods. The experiments showed that the proposed method often matches or outperforms existing OPE algorithms. 

main_review:  
The paper is written with exceptional clarity, where the authors provided both theoretical and experimental results backing their claim. Since there is no prior work on hyperparameter-less OPE, the authors chose a reasonable way to compare with other OPE baselines, where the baseline hyperparameters are finetuned on one set of environments and evaluated on another. The proposed method also took into consideration practical cases, such as 1. the state or action space is continuous 2.$(\pi, Q)$ pair is not trained till the optimum, where either the policy is not greedy w.r.t. Q, or the Q is under-trained before the policy converges.  
 
This is a significant work that will allow OPE to have more wide-spread use in practice where hyperparameter sweeps are expensive or infeasible. It is novel both compared to BVFT and to other OPE algorithms, providing both theoretical guarantees and practical flexibility (e.g. giving the option of BVFT-PE and BVFT-PE-Q depending on whether the user is able to tune one extra hyperparameter either online or on a similar environment.)
 
Minor nit-picking:
1. l88. ... to test more than 1 policies -> … to test more than 1 policy
 

limitations_and_societal_impact: The authors discussed the limitations of their method and either remedies or future research directions in the paper.
review_title: null

Meta-review: The manuscript examines the question of how to improve policy selection in the off-line RL setting. Typically offline policy selection is approached via off-policy evaluation (OPE), aimed at estimating the expected return of candidate policies. OPE is itself a difficult problem that typical requires hyperparameter tuning and selection itself. The paper develops moves closer to a hyperparameter-free method and demonstrates the effectiveness of the algorithm in the context of standardized offline datasets (e.g. RLUnplugged for Atari).
The algorithm for policy selection is built using insights from the recently published Batch Value-Function Tournament (BVFT) approach to estimating the best value function from among a set of candidates.
They make comparisons to well developed OPE style methods such as fitted Q-evaluation and show clear advantages in data efficiency and the policy selection.
The manuscript examines applying the approach to a wide range of settings (from Atari to continuous control) and to a range of policies produced by a variety of algorithms. The ideas, theory, and experiments are well motivated by the text. Taken together, the manuscript provides a promising look at a fundamental and open problem in RL. 

Decision: Accept (Poster)