Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper presents a framework for evaluating offline reinforcement learning (RL) algorithms.  Results from a  thorough series of experiments are presented which suggest that certain details of recently proposed RL methods are not necessary for achieving strong performance.  These results suggests that some of the complexity in RL design can be ignored.  

I commend the authors for performing a valuable test and comparison of existing offline RL methodology.

This paper could be improved by providing more clear insight and intuition about the deeper meaning of these results regarding the "unnecessary" technical complexities.  Could the authors suggest why certain complexities are unnecessary?  Clearly the authors of those previous works thought they were needed.  To really help researchers design better algorithms, we need to be guided by some insight about not only what doesn't work but why it doesn't work.

Also, the paper could be improved by being more clear about the nature of the evaluations.  The authors provide extensive results; but it wasn't clear whether these were "apples-to-apples" comparisons with the previous results in the papers that proposed the "unnecessary" technical complexities.  For example, I didn't see the authors say that they reproduced the results of previous works, only that they tested previous methods in certain tests.  Does the BRAC framework reproduce the results for previous papers?  If so, this should be made more clear and stated prominently in the paper so that the reader knows that BRAC is, in this reproduction of previous results sense, reliable.  If not, how if the reader to know that the "unnecessary" technical complexities, are truly unneccessary?

Finally, the paper presents lots of results, but I did not see any mention of the statistical significance of these results.

Minor issue:  In the Conclusion Section, the authors say, "Unfortunately, off-policy ... is an challenging open problem."  Unfortunately?!  A challenging open problem is a good thing!  And I think the authors did a good job addressing a difficult problem.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The paper introduces a general framework for behavior regularized actor-critic methods, and empirically evaluates recent offline RL algorithms and different design choices. 

Overall, the paper is well written and easy to follow.  I appreciate the authors for their careful empirical study. I am leaning to accept the paper because (1) the experimental design is rigorous and the results provide several insights into how to design a behavior regularized algorithm for offline RL.

There are some comments for the experiments.
1. Are the results significant (e.g. Figure 3 and 4)? Have you checked the error bars?
2. Missing numbers: trained_alpha in dataset 0 of Hopper-v2 in Figure 1 and SAC in dataset 0 of Hopper-v2 in Figure 6? Are they negative so not reported in the figure or just missing? 
3. Do you think the conclusion will change if you use training datasets of different size (e.g. much less than 1 million)? 
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper proposes a unifying framework, BRAC, which summarizes the idea and evaluates the effectiveness of recently proposed offline reinforcement learning algorithms, specifically BEAR, BCQ, and KL control. The authors generalize existing offline RL approaches to an actor-critic algorithm that regularizes the learned policy such that it can stay close to the behavior policy. Based on prior work, the authors state that there are two variants of regularizations to the behavior policy, value penalty (vp) and policy regularization (pr) and three choices of divergence functions along with their sample estimate that measures the distance between the learned policy and the behavior policy. The paper empirically investigates the effectiveness of each regularization scheme as well as each divergence function and conclude that vp is slightly more effective than pr while all divergence functions have similar performances.

Overall, this paper could be an interesting summary of prior works in offline RL and provide some empirical insights on the effectiveness of each building block in the previous approaches, though it neither offers theoretical explanations nor proposes a new offline RL algorithm that outperforms the existing methods under the BRAC framework. My score would be increased given some technical insights or some promising results in a relatively novel offline RL algorithm in the author’s response. 

Specifically, the paper does a thorough ablation study on BEAR, BCQ, and KL-control within the BRAC framework. While the experimental results demonstrate some interesting phenomenons such as combining vp and the primal form of KL divergence achieving the best performance and taking minimum over Q functions outperforming using a mix of maximum and minimum over the Q functions, I believe the paper would be greatly improved if the authors can provide a new offline RL method based on the BRAC that can achieve better performance than current approaches and is less incremental than simply combining vp and KL divergence. 

Though BRAC summarizes offline RL methods in a neat way, it would be more technically sound if a general theoretical analysis/insights of offline RL algorithms can be offered in the paper, e.g. showing the reason that vp is outperforming pr through convergence analysis in the tabular case.

Minor comment:
Error bars should be added to the all the bar plots.

**UPDATE**
After reading the rebuttal, I think my concerns are addressed and thus I updated my rating.
review_title: Official Blind Review #1

Meta-review: This paper is an empirical studies of methods to stabilize offline (ie, batch) RL methods where the dataset is available up front and not collected during learning. This can be an important setting in e.g. safety critical or production systems, where learned policies should not be applied on the real system until their performance and safety is verified. Since policies leave the area where training data is present, in such settings poor performance or divergence might result, unless divergence from the reference policy is regularized. This paper studies various methods to perform such regularization. 

The reviewers are all very happy about the thoroughness of the empirical work. The work only studies existing methods (and combination thereof), so the novelty is limited by design. The paper was also considered well written and easy to follow. The results were very similar between the considered regularizers, which somehow limits the usefulness of the paper as practical guideline (although at least now we know that perhaps we do not need to spend a lot of time choosing the best between these). Bigger differences were observed between "value penalties" versus "policy regularization". This seems to correspond to theoretical observations by Neu et al (https://arxiv.org/abs/1705.07798, 2017), which is not cited in the manuscript. Although unpublished, I think that work is highly relevant for the current manuscript, and I'd strongly recommend the authors to consider its content. Some minor comments about the paper are given below.

On the balance, the strong point of the paper is the empirical thoroughness and clarity, whereas novelty, significance, and theoretical analysis are weaker points. Due to the high selectivity of ICLR, I unfortunately have to recommend rejection for this manuscript.

I have some minor comments about the contents of the paper:
- The manuscript contains the line:  "Under this definition, such a behavior policy πb is always well-defined even
if the dataset was collected by multiple, distinct behavior policies". Wouldn't simply defining the behavior as a mixture of the underlying behavior policies (when known) work equally well?
- The paper mentions several earlier works that regularize policies update using the KL from a reference policy (or to a reference policy). The paper of Peters is cited in this context, although there the constraint is actually on the KL divergence between state-action distributions, resulting in a different type of regularization.
Decision: Reject