Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper explores how to collect informative data for offline RL methods. Many curiosity-based methods are considered to explore the environment. Intrinsic Model Predictive Control (IMPC) approach is proposed to improve the performance.
strength_and_weaknesses: The idea and setting are novel and interesting. However, the method part is a little difficult to follow. the description of the IMPC is not clear.
1. What are the input and output of IMPC? How do you train it? What is the main contribution?
2. The experiments only choose MPO to verify the quality of the samples, which is not convincing enough. Since you propose a new framework and use the offline RL methods as a mechanism for evaluating exploration performance, I think more than 2 most popular methods should be used,
3. The paper does not give a deeper analysis of the data requirement of the offline RL. In other words, in order to get better performance, which kind of data is required by the offline RL methods? How to construct such a dataset?
4. There are some papers focusing on offline training and online fine-tuning to improve performance via fewer data. They also use exploration methods to guide online data collocation. Their methods can also be compared after slight modification.
clarity,_quality,_novelty_and_reproducibility: Code is not provided and detailed implementation is not very clear.
summary_of_the_review: Please see my questions above.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper studies offline RL from a less-studied perspective - collecting informative experiences. The authors investigate the task-agnostic setting based on curiosity-based intrinsic motivation methods. The authors propose the Explore2Offline framework, and conduct an extensive empirical study to investing the effect of data collection strategies. 
strength_and_weaknesses: ### Strengths
The paper is clearly written and easy to follow. The authors also conduct extensive experiments to investigate data collection in offline RL based on a number of intrinsic motivation based exploration algorithms to reveal interesting findings. 

### Weakness
My main concern for the paper is its novelty. Although I acknowledge that the paper studies offline RL from a less-studied perspective (data collection instead of policy learning that aims to address extrapolation error) which is very interesting, it seems that it is very related to [1] without enough discussion about the differences. Specifically, [1] also studies data collection in offline RL, and it is worth discussing the differences between them.

[1] Denis Yarats, David Brandfonbrener, Hao Liu, Michael Laskin, Pieter Abbeel, Alessandro Lazaric, and Lerrel Pinto. Don’t change the algorithm, change the data: Exploratory data for offline reinforcement learning, 2022.
clarity,_quality,_novelty_and_reproducibility: Please see my comments in the above section.
summary_of_the_review: The paper studies offline RL from an interesting and less-studied perspective (data collection instead of policy learning), and propose a new exploration agent, Intrinsic Model Predictive Control that have strong performance. The authors also conduct extensive and in-depth analysis and investigation in standard offline RL tasks with different qualities. My main concern is its novelty, which is quite related to [1], but without enough discussion about their differences.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents a study of a reinforcement learning setting consisting of a task-agnostic data collection phase and a task-aware offline optimization phases, on a set of continuous control tasks. Comparing the performance obtained when using different exploration criteria (combined with a new MPC-based method) in the data collection phase, as well as different dataset sizes, this work tries to draw conclusions about the features we should expect to find in offline RL datasets for a downstream algorithm to perform well.
strength_and_weaknesses: Strengths:
- The problem addressed in the paper is of paramount importance: increasing our understanding of the kind of datasets that are needed to reach good performance with offline reinforcement learning has the potential to both shape the design of future offline RL algorithms and to have practical implications;
- The proposed MPC-based exploration method, which could be of independent interest, stands out as a clean way (probably cleaner than the reactive counterpart) of comparing the different exploration criteria.
- Despite being probably already known to some researchers in the field, having clearly written down some of the conclusions about the influence of dataset size and reward distribution could be useful to practitioners.

Weaknesses:
- The paper only uses the CRR offline RL algorithm for evaluating the downstream effects of the construction of different types of datasets. Despite this is a start, the actual outlook of offline RL algorithms used in the community is very vast, and there is really no baseline as established as for in the online RL case. This severely limits the scope of the work, since the reader cannot know a priori whether the conclusions of the study only hold for that specific offline RL algorithm or are indeed more general than that;
- The paper determines dataset size is possibly the most important variable to predict the performance for a downstream offline RL algorithm. However, this could be a bit misleading: one could imagine having the same trajectory over and over again in the dataset and clearly increasing the dataset size without any increase in the downstream performance. Despite this is a trivial counterexample, I think it is really epistemologically important to pin down the effect of diversity, more than size, on the resulting policy, because that one is more likely to be the underlying cause of the observed performance increase.
- The comparison with previous and parallel work is not very insightful from the perspective of the reader. For instance, it would be nice to understand more the relationship with (Yarats et al., 2022), instead of just briefly mentioning its existence. Moreover, there is some work similar in flavour about pretraining (https://arxiv.org/abs/2106.04799): it seems important to have a discussion on how the study presented in this work is different compared to that.
clarity,_quality,_novelty_and_reproducibility: The paper is mostly clear, albeit some of the conclusions could be presented more directly.

The quality of the work could be improved (are the results only relevant for a single offline RL algorithm? Is dataset size really the importance variable?)

The work is relatively novel, despite it misses important discussion with respect to previous and parallel work in unsupervised exploration.

There seem to be no apparent reproducibility issue (3 seeds are not great, but I understand the possible computational constraints the authors might have).
summary_of_the_review: Overall, I appreciate the direction presented in this work, as well as the general methodology that has been attempted. However, I believe there are some serious drawbacks that might overly reduce the scope of the conclusions drawn in the paper, and thus for now I tend to recommend rejection.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary:
The paper studies how to coleect data for offline RL. The main approach consists of a task-agnostic data collection phase and a task-aware offline optimization phases, on a set of continuous control tasks. The paper proposed an algorithm called Explore2Offline based on this idea and conduct an extensive empirical study to investing the effect of data collection strategies.

Strength:
- The paper is clearly written and easy to follow. The authors also conduct extensive experiments to investigate data collection in offline RL based on a number of intrinsic motivation based exploration algorithms to reveal interesting findings.
- Despite being probably already known to some researchers in the field, having clearly written down some of the conclusions about the influence of dataset size and reward distribution could be useful to practitioners.

Weakness:
- Missing related works about reward-free RL with function approximations
- Reviewers are concerned about the presence of a single offline RL algorithm in all of the experiments presented in the paper. It might be necessary to consider different offline RL algorithms
- The comparison with previous and parallel work is not very insightful from the perspective of the reader. 
justification_for_why_not_higher_score: The paper studies an interesting problem about how to collect data so that it can useful for downstream offline RL tasks. Yet, several reviewers are concerning about the novelty as existing works may have studied similar approaches and settings. The most concerned part is the lack of comparison with other offline algorithms, which may produce different effects on the data size.
Decision: Reject