Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper diagnoses some pitfalls with complex models in offline RL and proposes a very simple baseline model which attains competitive performance. This minimalist approach involves adding a behavior cloning term to the TD3 objective and normalizing the states and makes its hyperparameter easier to tune by normalizing by the value.
main_review: This paper attempts to cut through the often-confusing tangle of current algorithms for offline RL by providing a more straightforward alternative. The proposed method is in fact simple and easy to implement. The scaling rule for its hyperparameter is the least intuitive component but remains much simpler than the methods it compares to.

Ablating the state of the art algorithms by removing their implementation adjustments is a sensible thing to do. This emphasizes the need for "test environments" which researchers do not have access to for hyperparameter tuning; this is especially pressing in offline RL as access to the environment should be considered expensive and methods may overfit to the specific benchmark dataset. This need remains future work.

Overall this is the kind of paper that I'm glad someone wrote. It does not have sensational results or a flashy method, but it points out how much of current offline RL work is light with no heat. It provides a baseline that is easy and fast to train and which should probably be the bar for new research going forward.

### Notes to the authors
* Missing "which" or other connector at the end of line 188.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper presents a simple offline RL method that only requires a few lines of changes in code. The authors add a BC loss as the regularization to the standard policy update objective while also normalizing the states. The authors also normalize the coefficient of the BC loss with the average Q values in the dataset to balance the policy update objective and the BC loss. Through experiments, the proposed method appears to perform comparably to the best-performing prior offline RL approaches despite being simple and requiring lower tuning complexity.
main_review: The paper is well written and easy to understand. The proposed approach focuses on simplicity yet it seems to perform comparably to baselines in the empirical evaluations. The proposed method is also easier to tune and therefore makes it less costly to evaluate the train policies online after offline training. I think the method is of reasonable significance to the field of offline RL.

However, I do have a few concerns, which I will discuss as follows.

First, it seems that the novelty of the method is a bit limited. The authors seem to directly adapt RL+BC to the offline setting except that they add the state normalization, which is also not new. The authors also didn't theoretically justify the approach. For example, the authors should show that the method can guarantee safe policy improvement and moreover enjoys comparable or better policy improvement guarantees w.r.t. prior approaches. Without the theoretical justification and given the current form of the method, I think the method is a bit incremental.

Moreover, the empirical evaluations are not thorough. The authors only evaluated the approach in simple mujoco environments in D4RL. It is unclear if the method can perform well more undirected multitask datasets such as antmaze and kitchen as well as more complicated manipulation tasks such as adroit in D4RL. It also seems that the method does not perform well on random datasets. Is this a major limitation? I also think the authors should add state normalization to all baselines to ensure fair comparison as the state normalization is not a new technique in RL.

Finally, I think the comparisons are not complete. The authors should also compare the method to more recent model-free offline RL approaches such as [1] and model-based methods such as [2,3], which attains better performance on random and medium-replay datasets.

Overall, given the above comments, I would vote for a weak reject.

[1] Sinha, S., & Garg, A. (2021). S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning. arXiv preprint arXiv:2103.06326.

[2] R. Kidambi, A. Rajeswaran, P. Netrapalli, and T. Joachims. Morel: Model-based offline reinforcement learning. arXiv preprint arXiv:2005.05951, 2020.

[3] T. Yu, G. Thomas, L. Yu, S. Ermon, J. Zou, S. Levine, C. Finn, and T. Ma. Mopo: Model-based offline policy optimization. arXiv preprint arXiv:2005.13239, 2020.

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors revisit addition of a behavior regularization term for offline reinforcement learning but with minimal change to existing RL approach, namely TD3. They showcase their approach in D4RL gym environments. The results are surprisingly good for such a small change and  are competitive with SOTA for the most part. 

main_review: 
**Strengths:**

S1: Overall, the proposed method is very straightforward, fairly intuitive, and the performance seems comparable with the presented baselines. 

S3: The experimental results present a compelling case for simplicity in offline RL and calls into attention the problem of high variance in offline-trained policies. 

S2: With a few exceptions (discussed below), the presentation quality is high. Figures are and tables clear/readable. Writing is clear.

S3: The codebase accompanying the publication is minimal by design and supports high reproducibility. 
	
**Weaknesses:**

W1: No free Lunch - The proposed algorithm while being simple seems vulnerable to noisy datasets with a large number of sub-optimal trajectories. It would be nice to see some experiments where for the given datasets of  "random" and "replay" tags are combined with "expert" datasets. This will reveal the effect of regularization in a dataset collected from  diverse set of sub-optimal policies. Datasets with noisy action spaces would also be an interesting addition. The paper fails to showcase these edge cases where the current simple algorithm might fail over the more complex offline RL approaches. 
	
W2: The choice of hyperparameter does seem to affect the performance of the algorithm, However the initial choice of 2.5 seems to be arbitrary. This is because even if the actions are normalized the regularization term is dependent on dimension of the action space. Moreover a simple addition of random dimension in the action space might affect the performance of the solution. Hence it is still not clear how to heuristically choose the value of lambda for a new problem. 


**Minor Clarity Issues/ Comments:**

CL1: It would also be interesting to see if the performance holds up for challenging tasks such as Ant-Maze. 

CL2: In figure 5 It would be easy for the reader if it is explicitly mentioned that the full algorithm equates to" TD3 + BC + NORM + lambda" and replacing "- Norm" with " TD3 + BC + lambda". (If I understood the figure correctly.) 
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a new, simple approach to offline reinforcement learning, based on two modifications on top of a standard online RL algorithm (TD3) - introducing a BC loss to the policy loss, and normalizing the state features in the datasets. The authors show that these modifications achieve state-of-the-art performance for offline RL in the MuJoCo domain, even when compared to prior methods which involve many more moving pieces.
main_review: The originality of the work is small, as the proposed method is a small change on top of prior work. However, the authors recognize this, and I can see that the proposed method (TD3+BC) will be a strong but simple to implement baseline for all future offline RL algorithms going forward, so I view the contribution of this paper as potentially valuable and significant. The fact that such a simple modification results in near state-of-the-art results is surprising and potentally means there is still a lot of work for improving offline RL algorithms going forward. 

The writing of the paper is good and easy to understand. It is clear what the motivation of the paper (to propose a simple, easy-to-implement but competitive offline RL method) and I believe that the work meets those claims. 

For experiments, I believe that among model-free methods, these are the strongest or close to state-of-the-art results, which is fairly surprising given the simplicity of the method. One concern I have with the results is that they are primarily done in the MuJoCo domain, which are some of the easier tasks contained within the D4RL benchmark. I would be curious to see what the performance of the method is on a task such as the AntMaze navigation task, since in principle the task requires one to use dynamic programming to stitch together multiple incomplete trajectories. This would be a more interesting excercise for an offline RL algorithm where BC would not likely be as beneficial as it is for the MuJoCo tasks.

Overall, I think this is a good paper, mainly on the premise that it a) seems to be an overlooked method by the community, and b) achieves state-of-the-art results by a small margin on the domains reported. I think the paper could be stronger if there was more analysis of the method given that it is so simple (such as a theoretical analysis of the modified objective - does it still result in a convergent algorithm in simplified cases such as tabular, does adding BC loss prevent the method from achieving optimal returns, etc.). However, I still think the current experimental results are in general satisfactory.


limitations_and_societal_impact: Yes, the authors discuss some limitations of the method with regards to stability of the learned policies, and potential negative consequences such as negative applications of offline RL and data bias.
review_title: null

Meta-review: Overall this paper makes a nice contribution and is arguably an example of the type of work our fields need more of. 

My only reservation is that the authors should have done more to highlight deficiencies of the approach, which may not be present for some of the other approaches. In particular, the approach obviously should suffer when data has coverage of good policies, but may not correspond to a good policy that one should aim to imitate. The authors are aware of this and will ideally make this point more strongly in the revision. Regardless, the paper still makes a nice contribution. 


Decision: Accept (Spotlight)