Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper intends to improve offline reinforcement learning (RL) with limited data from a data-partition perspective. The authors first analyze why traditional data splittings don't perform well, including non-splitting (on the entire dataset) and cross-validation splitting. Finally, the authors propose a repeating random sampling strategy to enhance the selection of algorithm-hyperparameter (AH). The authors design experiments to demonstrate the superiority of the proposed SSR-RRS method in terms of policy performance.
strengths_and_weaknesses: Strengths:
1. The research topic, improving offline RL from the data splitting angle, is inspiring and innovative. Data splitting is widely investigated in supervised learning, but it is rarely studied in the RL area. This paper proves that data splitting has a significant impact on offline RL training.
2. The idea of repeated random sub-sampling (RRS) is novel. The RRS strategy integrates ideas of ensemble learning and cross-validation, which is maybe not very fancy, yet very innovative and effective.
3. Almost all claims and theorems are justified either from the theoretical or the empirical analysis, making the paper solid and sound.


Weaknesses:
1. The parameter sensitive analysis for the RRS method is not comprehensive. To make the work more practical, the readers would like to know how to choose a reasonable splitting number $K$. The experiments show that when $K=5$ has the best performance. Does this mean the larger $K$ the better? Will it drop when reaching some thresholds?
questions: 1. In line 37, the authors claim that the trained policies cannot be well evaluated, when the validation set has limited high-reward trajectories. This is doubtful, because the training set is responsible for the learning process, while the validation set is only used to 'predict'. It seems that after the policy is trained on the training set, its 'prediction' capability is not affected by the validation set. Is there any justification for that?
2. In RRS, can the policies trained by previous splittings be used as the pre-training for current splitting?
limitations: There is no apparent limitation on the work. It is universal and can be adapted to different offline RL algorithms. It will have a broad impact on the RL area.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a method for offline policy selection for offline reinforcement learning algorithms. The main idea is to use repeated random splitting of the dataset to pick the best hyper-parameters before training on the entire dataset. The algorithm is evaluated against multiple datasets and compared against standard benchmarks. 
strengths_and_weaknesses: Strengths:
- The paper tackles an important problem in offline reinforcement learning. Without a policy selection method, it is likely that the resulting policies perform poorly after deployment. 
- The algorithm proposed is well motivated with theory and compared with the related works in literature.
- The evaluation procedure is systematic, with datasets from multiple domains in reinforcement learning and comparisons against the state-of-the-art policy selection algorithms. 

Weaknesses:
- Some of the claims in the paper do not have evidence or reasoning. Details below. 
questions: Q1. The proof of theorem 1 assumes that the behavior policy is uniform random, and that high reward trajectories are sparse. This may not be true in practice. Historical data is usually from a non-random policy already deployed in a system. Can you explain why this assumption holds, and what are the implications if it is voided.

Q2. In line 174, you claim "WIS will return the observed return of the behavior policy if averaging over a single trajectory, independent of the target policy to be evaluated." It is not clear why this is true. There is no evidence, analysis or citation. 

Q3. Line 207 claims: "As K → ∞, RRS approaches the leave-p-out cross-validation (CV), where p denotes the number of examples in the validation dataset." Again, not clear why this is true. How does p relate to the size of the dataset you use in RRS (split by 2)? 
limitations: The entire paper relies on a single evaluation method for a dataset (WIS for discrete, FQE for continuous MDPs). It would be good to see if the claims hold for another evaluation method.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper studies the model and hyper-parameter selection in offline RL, which is an important problem as most offline RL algorithms are sensitive to the algorithm it used, the hyper-parameters and other subtle algorithm details. Different from supervised learning, OPE estimate on the validation set is not reliable when the sample size is small and the trajectory is long. This paper proposes SSR (Repeated Random Sampling) for algorithm selection, the idea is simple in which we just randomly split the trajectories as train/val multiple times, and perform algorithm training and OPE on the train/val separately. The algorithm configuration that achieves the highest average performance on the val will be selected, and finally re-train the whole dataset using this selected configuration. Empirical studies on different domains, such as healthcare, robotics show better performance compared with other baselines, such as BVFT, CV. 
strengths_and_weaknesses: Strength:

[1]. The paper takes a step towards an important problem in offline RL, i.e., the hyper-parameter selection. As most offline RL algorithms are very sensitive to this and it is important to understand the rule/pipeline for how to effectively select the best configuration for the training algorithm.

[2]. The method proposes is simple and well-motivated, especially on how it compared with K-fold CV. 

[3]. The ablation study is well-done, especially on the importance of re-train procedure, the impact of the repeats training and the effect of the datasize.

Weakness:

[1]. This method somehow sounds like more of a heuristic. Though Theorem 1 tries to show there exists a setting that this k-split average performance is better than the 1 split, it does not show the correctness of the method, and this result seems weak for the theoretical justification. 

[2]. If I understand it correctly, the computational burden for the method seems high, as it needs to retrain for K times, and the training for RL is very time-consuming.

[3]. The paper frames it as an overall pipeline, it discusses more on train/val split, however for the OPE estimate part is missing. How is the pipeline sensitive to the OPE estimate used? Is there any ablation study on this? In the experiments, it just uses the default setting, WIS for discrete and FQE for continuous task.

questions: [1]. I appreciate the effort on making the pipeline algorithm agnostic. One thing for further understanding this, does the author observe any correlation about the training algorithm used and the OPE used for selection? Say is FQE performs better on FQI algorithms in terms of selection compared with WIS? Is there any interplay on this?

[2]. The empirical study tries to span different domains, which is very nice. It would be very helpful if the authors try to organize them in a way that shows the properties of these MDP/POMDP problems, such as sto/det reward, transitions, sparse/dense reward, horizon length etc and the dataset composition: narrow or diverse. It might be interesting to see how the method works across these different settings, and provides some practical recommendations

[3]. Any commends/ideas on reducing the time complexity of the algorithm, as it involves training for multiple times.


Minor:

[1]. What are the y-axis in Fig 4(a)? It seems we have 540 AH pairs, but the y-axis is much smaller than this.
[2]. Is there a typo on the legend of Fig 4(b)? SSR-RRS 5 is missing.
limitations: Yes.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes SSR, a new pipeline for selecting the best-performing deployment policy in the offline RL setting. The key idea of SSR is the use of repeated random sampling (RRS), which samples $K$ overlapping train-test pairs from the offline dataset. Each specific algorithm-hyperparameter choice is trained and evaluated $K$ times using these pairs. SSR picks the policy with the highest average performance. The authors justify SSR both theoretically and empirically. SSR consistently outperforms other policy selection methods over various environments, especially in low-data regime.
strengths_and_weaknesses: ### Strengths
- The paper tackles the problem of hyperparameter tuning, or policy selection, which is a relevant and essential aspect of offline RL.
- To the best of my knowledge, the idea presented is original. While repeated random sampling is not a new technique, it has never been tried in the offline RL setting.
- The idea is simple and well motivated. The authors did a good job of pointing out the main limitation of current OPE and OPL methods: the high variance with respect to the train-validation split. SSR naturally solves this issue by performing this split multiple times and averaging the performance.
- The authors evaluate SSR on extensive benchmarks, ranging from robotics to medical domains. SSR consistently produces better policies than other selection methods.
- The paper is easy to read in general.

### Weaknesses
- My first concern regards the robustness of SSR with respect to the chosen offline RL algorithm. In the paper, the authors only show the performance of the best policy among all AH pairs. However, it is also interesting to see if SSR consistently performs well given a specific algorithm. This reduces policy selection to hyperparameter tuning, which is important for future research in offline RL.
- The second concern regards the effectiveness of SSR with respect to the choice of OPE. Will there be an OPE that is robust enough to the partitioning of the dataset that using repeated sampling will not help? I would love to see at least one more OPE being used in SSR in addition to WIS or FQE.
questions: - Take computation aside, will increasing $K$ always improve the performance of the final policy?

Minor questions/suggestions:
- Legend of Figure 4(b) is confusing, is there a typo there?
limitations: The authors have adequately addressed the limitations of the work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a workflow for hyperparameter and algorithm selection in offline RL. The authors argue that because datasets can be small and high variance, and algorithms themselves are hyperparameter sensitive, the prevailing approach of training on a fixed training set and tuning on a validation set is not sufficient. The authors instead argue to use re-sampling of the dataset and then average OPE estimates on the validation set in order to estimate the performance of the algorithm & hyperparameters. The authors evaluate on a collection of tasks (based on D4RL, MIMIC-III Sepsis, Robomimic, and Tutorbot) where they show their selection method typically finds the best policies compared to the baselines.
strengths_and_weaknesses: - The evaluation is well-done and convincingly shows the utility of the proposed approach. The authors used several logical baselines and several domains that are fairly different qualitatively (ranging from healthcare to robotic control). 
- This work has potential to be fairly significant within the field of offline RL, since it is applicable to most algorithms, and performs better than the common naive approach to hyperparameter selection.
- The writing is clear and well-organized, The authors motivate the use of their method from theory, describe their practical approach, and clearly explain the rationale behind their chosen baselines.

questions: - What was the computational cost of the various approaches used in evaluation? While SSR-RRS 5 produces the best results, does it also require significantly more compute that other approaches? 
- My intuition is that OPE is fairly inaccurate in predicting the values of the policies, esp on high-dimensional domains. Was the accuracy of OPE a bottleneck for the method, or did it general predict the relative ordering of policies correctly?
limitations: The authors did not discuss potential negative societal impacts of their work, but they discussed a reasonable set of limitations for their method.
review_title: null

Meta-review: metareview: ## Summary
In offline RL, typically it is not possible to run the policies to evaluate them in the environment for model selection, as a result one needs to rely on the offline approaches such as OPE methods to do the model selection. This paper points to the importance of data partitioning over single partitioning for offline model and hyper parameter selection. They propose a simple but effective method called SSR for this purpose and compare non-splitting and different ways of cross-validation splitting. The paper has comprehensive experiments on D4RL, Robomimic, Tutorbot and Sepsis domains. They show very promising results on those tasks over other approaches for offline policy selection.

## Decision
Overall this paper is very well-written and studying a very important problem for offline RL. The paper was already in a good shape when submitted, however, after a few small points made by the reviewers I think it will be in a much better shape. I think the offline RL and NeurIPS community would benefit from the findings in this paper and deserves to be accepted. 

I would still recommend the authors to clarify the points that authors were confused in the paper as they did in the rebuttal and include some of the results presented in the rebuttal in the main paper.



confidence: Certain
Decision: Accept