Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This paper proposes a Near real-world offline RL benchmark, named NeoRL. The NeoRL dataset is collected on different real-world tasks with a more conservative strategy compared with the previous offline RL benchmark. They evaluate some SOTA offline RL algorithms on NeoRL and reveal that current offline RL algorithms are less effective in real-world tasks. The authors propose a new workflow for offline RL policies: offline training, offline validation (new), and online deployment. 
strengths: The NeoRL benchmark contains sufficient datasets since it involves 7 domains with 52 tasks in total. Besides, the authors build benchmarks for offline and online policy selection. Moreover, they conducted sufficient experiments to verify the performance of existing offline RL algorithms. And they pointed out some interesting results which can give the right direction for developing offline RL policies in real-world tasks.
weaknesses: For the NeoRL benchmark, the authors claim that limited samples (1e4) if enough for policy training. However, more training data is helpful to train a more robust policy, especially when we take the security issues of RL policy into account (e.g., adversary attacks).

For the offline validation process, the proposed validation with offline data, before a policy is deployed on a real-world task, indeed improves the efficiency of policy selection. But from Table 1, I noticed that the average rank in online performance is different from the rank in offline performance (e.g., CQL). I am wondering can the offline validation reflect the correct performance rank of a policy in the real-world task? And how to prevent selecting a sub-optimal policy?

correctness: The claims in this paper seem correct. The evaluation methods are appropriate. 
clarity: This paper is well written and easy to follow. 
relation to prior work: This paper well discussed previous works about RL benchmarks and Offline RL algorithms in section 3.
additional feedback: Typos: 
-	Line 87: redundant period
review_title: paper review

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This paper presents a new offline RL benchmark that considers using a conservative working policy (i.e. deterministic) for data collection in various tasks that simulate real-world applications and also tests the performance of offline evaluation methods. The results show that current offline RL methods don't work well in the benchmark compared to BC, suggesting more improvement is needed.
strengths: 1. It is valuable to consider more conservative data composition in offline RL, which matches the need of many real-world tasks.

2. The authors compare the performances of current methods under online evaluation and offline evaluation respectively. This is of great value to the community since performing offline evaluation is pivotal in offline RL research.

3. The tasks constructed in the paper are closer to real-world applications.
weaknesses: 1. The conservative data collection part is actually very similar to the medium data in previous benchmarks, e.g. D4RL, where a fixed policy collects the data with medium-level performance. The paper instead proposes to run the mean of the fixed Gaussian policy 80% of the time and the stochastic Gaussian policy otherwise. I'm not sure if such a change can lead to more practical applications and make a huge difference in the performances of offline RL algorithms.

2. While I generally agree that we need to be more conservative in data collection, I don't think that means we should only collect data with a fixed mostly deterministic policy. Such a setting proposed in the paper could make the data coverage very narrow, thus benefitting BC and model-free methods. I think it would be worth studying the dataset that consists of a mixture of policies or random policies that don't violate certain safety criteria. Those settings should still be practical in real-world tasks and could result in very different outcomes. 
correctness: The evaluation methods and experimental design seem correct. The claim that the real-world datasets should be collected with a deterministic working policy seems a bit off (see the section above).
clarity: Yes
relation to prior work: Yes
additional feedback: See the weakness section.
review_title: Official Review

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: Previous offline RL benchmarks commonly involve significant reality gaps, which include rich and overly exploratory datasets, degraded baseline, and missing policy validation. This paper proposes NeoRL, a suite of Near real-world benchmarks for offline RL. 
strengths: 1. The authors collected a lot of real world data, including locomotion control, industrial control, financial trading, sales promotion and city management domains with real-world dataset properties.
2. To investigate whether these offline RL algorithms are effective, the author compare them with the working policy.
3. The author adds a section on offline policy validation, which feels inspired.
weaknesses: 1. Why did you only choose these three environments? I see that the commonality of these three environments is relatively simple. HalfCheetah-v3, Walker2d-v3, and Hopper-v3. Why not choose Humanoid, whose state space 376 and action space 17.
2. I have great doubts about the author's point of view, and I hope the author can further elaborate on this point of view. 
    “Comparison with the working policy.The experimental results show that they appear less effective when compared with the working policy.”


correctness: All my concerns have been mentioned above.
clarity: This article is well written and illustrates the author's ideas very clearly.
relation to prior work: The paper makes good connections to many prior works.
additional feedback: I has presented it in the previous section.
review_title: This article is well written and many experiments have been designed.

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper presents a near real-world offline RL benchmark (NeoRL). It concentrates on learning from conservative and limited data, because many real-world working policies are conservative. In addition, it evaluates the offline policies in both offline and online ways, where offline evaluation is more reasonable in a real-world application. The authors investigate popular offline RL algorithms in the near-real setting and find that many offline RL algorithms may be overestimated.
strengths: I think the topics of learning from conservative data and offline evaluation are very meaningful. They are important in deploying real-world RL applications.

Since many real-world policies cannot be too exploratory, I agree with the authors that a good offline policy should perform well on conservative data. The results show that many offline RL algorithms may be overestimated, which will inspire future works.

The paper is written clearly.
weaknesses: The authors propose a complete pipeline for deploying offline RL in Fig.1, but they don't follow such a pipeline to evaluate the RL algorithms. Instead, they show the results of online evaluation and offline evaluation respectively. And the results show that online evaluation and offline evaluation could not reach an agreement. It is confusing if the policy selected by offline evaluation is better than the deterministic working policy.

correctness: It seems correct to me.
clarity: The paper is well written.
relation to prior work: The authors discuss the relation to prior works sufficiently.
additional feedback: 1. I think the authors could try some more offline evaluation methods.

2. I think it is better to follow the pipeline proposed in Fig.1 to evaluate the offline RL algorithms, including offline selection, and evaluate the best policy online.
review_title: A meaningful benchmark for deploying real-world RL applications.

final_score: rating: 10: Top 5% of accepted papers, seminal paper
review_content: summary and contributions: In this paper, the authors propose a novel benchmark for offline reinforcement learning, NeoRL. NeoR includes real-world tasks, e.g., industrial benchmark, FinRL, CityLearn, SalesPromotion. The datasets in NeoRL are conservative and limited, which follows the properties of real-world datasets. The authors propose a new method to evaluate the performance of offline RL algorithms. Instead of comparing with the behavior clone, they propose to compare with the working policy. The benchmark introduces an offline validation phase, which evaluates the learned policy using a test dataset before deploying it to the environment. A lot of existing offline methods are evaluated in the benchmark. NeoRL is more practical and challenging compared with previous benchmarks, and I believe that it will become a popular benchmark for offline learning.
strengths: 1. NeoRL includes diverse datasets of real-world tasks, which are more conservative and limited than the previous ones.

2. NeoRL introduces offline validation in the offline training pipeline, which is not formally considered in the previous ones.

3. To clearly investigate the effectiveness of offline RL algorithms, NeoRL compares the performance of learned policy with working policy, rather than behavior clone policy.

4. Based on the proposed pipeline of offline training, the authors re-evaluate many existing offline algorithms.
weaknesses: I have a question about policy evaluation. In CV tasks, researchers will select the best model in the validation set. However, in reinforcement learning, the convergence is the most important, and researchers usually care about the performance of the converged policy. We know the best policy might not be the converged policy. So, in offline RL, should we focus on the best policy or the converged policy?
correctness: I cannot see any technical incorrectness in this paper.
clarity: The paper is well written
relation to prior work: The relation and difference between this work and previous offline benchmarks, e.g., D4RL, are very clear.
additional feedback: We know offline RL is a fast-growing field, there are many existing algorithms. More baselines are expected to be evaluated.
review_title: Review

Meta-review: metareview: The paper introduces new datasets and benchmarks for offline settings in RL. The paper's contributions are notable for (1) using and highlighting the importance of learning from narrow data distributions, (2) treating policy evaluation and validation as a key challenge rather than an afterthought, and (3) providing extensive results of existing methods. I believe this paper is a significant contribution to the field and hope that future researchers will use it in addition to the existing offline RL benchmarks.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept