Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary: 
This paper focuses on the problem of Q value over-estimation in offline reinforcement learning and proposes three approaches (tricks) to help solve this problem. (1) estimate Q value of behavior policy avoiding max-operator in Q learning and take greedy action according to the behavior value estimation. (2) introduce ranking loss to push down the value estimation of all unobserved state-action pairs to avoid over-estimation. (3) use tanh operator to bound the range of Q value estimation, and learn a scale parameter with regularization term. The experimental results on several domains (Atari, Bsuite, Deepmind Lab) with discrete action space show performance better than existing algorithms.

Clarity:
This paper is generally written clearly, though I have several questions about the technique and experiments, which may need more clarification. Please see 'Cons' part for the detailed questions.

Originality:
The techniques of ranking regularization and re-parameterization of Q-values are novel in the literature of offline reinforcement learning. Behavior value estimation removing the max operator to alleviate over-estimation seems not novel, because many previous work in offline RL (e.g. SPIBB, ABM, CRR) use bellman operator without max operature for policy evaluation of target policy. Also, the one-step policy improvement in section 3.1 seems not novel, many previous work (e.g. BEAR, BCQ) sample the action according to the learned policy and take one of the sampled action with maximum value estimation at test time in the implementation.

Significance:
The main concern of the proposed method is whether it is theoretically sound and performs well in the other domains (such as domains with continuous action space) without much tuning of hyper-parameters (weight of regularization term when combining the proposed approaches). The three tricks are intuitive and might be useful in practice, but I am not sure whether the contributions are significant enough to match the acceptance bar of ICLR.

Pros:
* This paper attempts to solve a significant problem (extrapolation error in offline RL).
* The paper explains the intuition behind each proposed approach clearly.
* The experimental results are good on several domains with discrete action space, better than the baseline methods.

Cons:
* In section 1, "Surprisingly, this technique with only one round of improvement ... often outperform existing offline RL algorithms" seems a bit misleading and overclaiming. The proposed technique (1) can be better than behavioral policy only when the behavior policy is not deterministic and greedy with respect to the value estimation. And the experiment only verifies that it can outperforms the existing methods in some specific datasets collected on domains with discrete action. I doubt whether this technique can "often outperform" the existing algorithms (e.g. ABM, CRR, CQL) on continuous control tasks.
* Overall, the proposed techniques (2) (3) are not quite convincing without the support of the theory. In equation (5), the specific formulation of the weight of regularization, e.g. $exp((G^B(s)−E_{s∼D}[G^B(s)])/β)$ is not well-motivated. Why we use exp function instead of another simpler monotonically increasing function? Why we need the coefficient β? How to choose the value of ν and β (the given value in the paper are just randomly picked)? In equation (7), the regularization for learning the scale parameter is not well-motivated either. Is it really necessary to have this regularization? Why square function here is better than other functions? Without a theoretical ground, all these design choices and value choices are just like heuristic or magic numbers. The experiments show these choices can work on some dataset (perhaps with much tweak and tuning), but we are not confident whether they can also work on new datasets.
*In section 4.1 and 4.2, the performance of the proposed method is not significantly better than the baselines (the error bar of the proposed method overlaps with the error bar of the baselines).
*In section 4.1 and 4.2, QRr and BRr are mainly considered, but in section 4.3, only B and BR are considered. I am curious whether BRr and QRr perform well in this domain? If not, could you please explain why different combinations of the three propose techniques perform differently in different domains? Is there any principle about which combination should be used in which kind of dataset?
*In Figure 10 and 11, it seems that on Atari games, the larger weight of the ranking regularization means better performance. Then why "0.05 seems to be the optimal choice for the ranking regularization hyper-parameter"? Is the hyper-parameter value 0.05 used across all the datasets? Is the proposed approach sensitive to this value?
*In Algorithm 1, is it a typo of using γ? In the main text, γ means the discounting factor, then what's the definition of $\mathcal{L}(\gamma)$? The details of the re-parameterization of Q network need more clarification.

review_title: Good empirical results but may be difficult to work in other domains

final_score: rating: 4: Ok but not good enough - rejection
review_content: ##### Summary & recommendation 
The paper proposes an offline RL algorithm, which consists of three techniques: behavior value estimation, ranking regularization, and reparametrization of Q function, to reduce overestimation errors. The algorithm is evaluated on several benchmark datasets. 

Overall, the paper is clearly written. The empirical results also seem promising. However, my main concern is that the proposed algorithm, especially with the behavior value estimation technique, has a big limitation for solving the offline RL problem (details come below). Moreover, I don’t think the paper provide enough justification for using all three techniques other than experiment results. It would have been better if the authors could discuss why we need all these three techniques (e.g. maybe behavior value estimation + ranking regularization is similar to behavior regularization?), rather than just combing these tricks to make the algorithm work empirically. I think there is still room for improvement before publishing this paper. Therefore, I recommend to reject the paper.

##### Supporting arguments
The goal of an offline RL algorithm is to find a nearly optimal policy $\pi$ from an offline dataset. However, the behavior value estimation technique is just learning the value function for the behavior policy. Even though we perform a single policy improvement step in the test time, it is generally not sufficient to obtain a nearly-optimal policy, especially when the behavior policy is far from optimal. 

The authors mention “Fortunately, this one step is typically sufficient for dramatic gains as we show in our experiments (see for example Fig. 9). This finding matches our understanding that policy iteration algorithms typically do not require more than a few steps to converge to the optimal policy (Lagoudakis & Parr, 2003; Sutton & Barto, 2018, Chapter 4.3)”. I think this is not true. Even in tabular case, policy iteration requires a polynomial time w.r.t. the size of the state space and the action space (e.g., see Theorem 1.14 of [1]). I think it is possible to construct a family of MDPs and some behavior policies such that one step of policy improvement is not sufficient. In such case, I think the proposed algorithm would not work well. 

The related work section should not just list previous works, but explain how the proposed algorithm is different from or similar to the existing algorithms. 

I have some questions to clarify my understanding of the paper: 
* I am not sure what is the propose of Appendix A? Maybe I missed some important points here, but it has already been shown that off-policy + function approximation + bootstrapping can diverge (Sutton and Barto’s book).
* What is the exact definition of extrapolation error used in this paper? The paper mentions extrapolation over-estimation, extrapolation under-estimation, training time extrapolation error, and testing time extrapolation error, but I don’t see a clear definition of these terms. 
* Regarding the experiments: How were $\nu$, $\beta$, and $\alpha$ selected? The algorithm introduces more hyper-parameters, so I wonder do you have any comments on hyperparameter search (e.g. do existing algorithms also require tuning extra hyper-parameters?). Do you have any reason why it needs a larger mini-batch and a smaller learning rate to update $\alpha$? 
 
##### Minor comments
* $\theta’$ in Equation (2) is not defined 

[1] Alekh Agarwal, Nan Jiang, Sham Kakade, and Wen Sun. Reinforcement Learning: Theory and Algorithms. 

review_title: Promising experimental results but the algorithm has some limitations  

final_score: rating: 3: Clear rejection
review_content: Summary:
The paper deals with offline aka batch RL for discrete actions. Three techniques ((i) behavior value estimation, (ii) ranking regularization, and (iii) reparametrization of the value function), which can be combined with each other, are presented. These techniques are compared with other methods in different experiments. Furthermore a new benchmark is being introduced.  It is claimed that in this new benchmark, the new techniques outperform state-of-the-art methods. Furthermore it is claimed that the presented method „behavior value estimation“, although it is only a one-step greedy optimization is typically already sufficient for dramatic gains.

Strong points:
The abstract and the first part of the introduction (the first 1.5 pages) are very well written and the problems of offline RL are very well presented. Also very good is the consideration that the existence of a behavior policy is a restriction that does not apply to every given dataset, as expressed in the terms "behavior policy(s)" and "coherent policy". However, it is not specified in the text what exactly is meant by "coherent policy".

Weak points:
The representation becomes increasingly unclear from page 2 onwards. None of the three techniques presented is sufficiently discussed and sufficiently tested. None of the statements is supported convincingly, although the paper already makes extensive use of references to the Appendix. There are 14 references in the main text to the Appendix and four to figures in the Appendix.

Recommendation:
In its current form, the experimental part of the paper is immature. A uniform structure is missing. The statements are not sufficiently substantiated. Therefore I recommend to reject the paper. It seems that there is not enough space to present and sufficiently verify all three techniques.

The claim "this one step is typically sufficient for dramatic gains as we show in our experiments (see for example Fig. 9)" is not sufficiently substantiated, because one cannot speak of "typically", as only "Atari online policy selection games" are considered. And additionally Fig. 9 is located in the Appendix.
The meaning of the error bars in Fig. 3 and Fig. 7 is not explained.
The (on first sight) counterintuitive result that the performance of CQL and QRr at cart-pole is higher at 40% noise than at 0% must be explained in the text or caption.
Because the presented ideas are not sufficiently examined and supported, no sound knowledge is generated on which the reader can rely.

Questions:
What is the meaning of error bars in Fig. 3 and Fig. 7?
What is meant by "BC"?

Additional feedback with the aim to improve the paper:
The abbreviation BC is not introduced. It is unclear what is meant by it.
It is unclear what is meant by "coherent policy“.
What is meant by "discrete offline RL algorithms“?
it is not clearly described that discrete actions are required.
In the sum, i runs from 0 to 100 and is divided by 100, but from 0 to 100 we count 101 (unfortunately there are no line numbers in the manuscript, to locate the sum).
"5e - 2" does not look nice, better is 0.05 or $5 \cdot 10^{-2}$.
In Figure 8, the measurement results should not be connected by lines. Lines should only be used for fits or predictions of theory.
In Appendix F the text width is not respected.	
Please do not use \pm for the standard deviation, but for the specification of the uncertainty (aka error of the measurement) e.g. the standard error.

-----------------------------------------
(Dec 3.) Although I appreciate the feedback, my assessment of the paper remains unchanged.





review_title: no sound knowledge to rely on

Meta-review: This paper proposed a new method for improving offline RL. AC thinks that the paper has a potential, but all reviewers suggest rejection as the current write-up is quite poor. This causes many misunderstandings of reviewers. The authors clarify some misunderstandings/concerns in the discussion phase, but did not update the draft accordingly. Hence, AC cannot suggest acceptance, given the current form.
Decision: Reject