Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper considers a setting where we are given access to a dataset of expert or noisy-expert data collected from some MDP and need to decide whether to use either behavior cloning (BC) or offline RL. It conducts a theoretical analysis in a tabular setting showing that offline RL will recover a better policy than BC when the data is sufficiently suboptimal and has sufficient coverage. Finally it conducts an empirical analysis that confirms the results in some diagnostic gridworld tasks and shows that a tuned variant of the CQL offline RL algorithm outperforms BC in several larger-scale tasks.
main_review: ### Strengths

1. The writing is clear throughout. The paper clearly explains its goals and is well organized. 
2. The proofs seem to be generally correct. 
3. The empirical analysis is relatively thorough and well documented. It seems to generally follow best practices with respect to tuning hyperparameters, reporting confidence intervals, and even reporting IQM on the Atari experiments. 

### Weaknesses

1. I'm somewhat skeptical of the usefulness of the question the paper is trying to answer. First, from a practitioners perspective it is not clear why I have to choose between BC and offline RL. I can always run both and use some sort of validation procedure to decide which policy to deploy. Second, it seems to also be a false choice since offline RL algorithms often contain a hyperparameter that reduced the algorithm to BC if set to an extreme value, and in this sense BC is strictly contained in the offline RL algorithm. Third, the paper does not seem to consider the standard cases where BC is desirable and offline RL may not even apply. For example, BC is often used in problems where there is no obvious reward function (e.g. in many robotics problems), but we can get demonstrations of the desired behavior (e.g. from tele-operating). Finally, I am worried that the paper is arguing against a bit of a strawman. I am not aware of anyone making the argument that BC should be preferred to offline RL when a reward function and sufficient data are available. 
2. The theory only considers tabular representations and as a result avoids the central issues of offline RL. By limiting to the tabular case, the analysis does not have to deal with the notorious deadly triad. Wang et al., 2021 and Zanette, 2020 (cited in the paper) prove exponential lower bounds on sample complexity in the case of linear function approximation. Importantly the results depend on errors due to the function approximation. Empirically, Brandfonbrener et al., 2021 (also cited in the paper) bears out some of these concerns caused by amplifying approximation errors in offline RL. By assuming away this key part of the problem, it is not surprising to see positive results for offline RL. 
3. I am somewhat worried about novelty. On the theory side, the positive results largely exist in prior work and are even more general since guarantees exist beyond the tabular setting already. The comparison of the results across algorithms may be novel, but the results themselves do not seem to be a major contribution. On the empirical side, the sorts of results presented already exist even in the original paper that proposed the D4RL benchmark, showing that BC outperforms CQL sometimes and then in the paper proposing tuning methods for CQL demonstrating the improved performance from tuning. The emphasis on this comparison may be novel, but the results themselves do not seem to be.
4. I couldn't find any details at all about the BC-PI implementation used in the experiments. Could the authors comment on how this algorithm was implemented? This is especially interesting to me because it seems that a naive algorithm that attempted to filter out the bottom half of state-actions from the dataset would work very well on a dataset that is just half random and half expert data, so I'm wondering if there is something strange going on with the implementation. 
summary_of_the_review: While the paper is generally clear and technically sound, I found several serious issue with it and recommend rejection at this time. Specifically, I am skeptical of the usefulness of the central question, worried that the theory is assuming away the interesting part of the problem, worried about novelty, and had a few more minor concerns about the experimental setup.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Offline RL approaches are of quite great interest because of potentially easier real world applications. But as the paper points out in the extensive related work, there are a lot of conflicting results in the literature when it comes to comparison with plain old behavior cloning. To this end, the paper functions as fairly clear exposition of the current state of offline RL literature and proposes a formalization to allow comparison between these approaches. This formalization leads to a better characterization for the conditions for offline RL methods to outperform behavior cloning (and the importance of planning horizon and critical states).

The paper performs experiments on a wide number of domains which allows the readers to make their own inferences for the conditions for offline RL successes.

main_review: **Strengths**

Offline RL approaches are of quite great interest because of potentially easier real world applications. But as the paper points out in the extensive related work, there are a lot of conflicting results in the literature when it comes to comparison with plain old behavior cloning. To this end, the paper functions as fairly clear exposition of the current state of offline RL literature and proposes a formalization to allow comparison between these approaches. This formalization leads to a better characterization for the conditions for offline RL methods to outperform behavior cloning (and the importance of planning horizon and critical states).

The paper performs experiments on a wide number of domains which allows the readers to make their own inferences for the conditions for offline RL successes.

**Weaknesses**

It's unclear how much bearing the theory has on the actual empirical experiments as the authors themselves emphasize the importance of tuning offline RL. Tuning techniques seem domain dependent and exactly what features of the domain lead to particular tuning choices remains unexplored. Nevertheless, since the tuning methods are offline, it's not a major issue. Overall, some way to separate the [state] representation learning problem from the best action problem is likely important to really figure out the problem which the theory in the paper largely ignores.

The bigger issue is whether the "noisy-expert" data is a good proxy for real-world offline RL applications. While the paper cites a few examples of suboptimal data collection in Sec. 4.3 for most real world applications, the data is actually seldom noisy in the same way. Often a more realisitic assumption is that there are different suboptimal demonstrators (consider different human operators with different skill levels) who perform the task in different ways but their policies can't be simply described as noisy versions of each other (or the highest skill level). Moreover, there are often many safety constraints that simply would not allow running certain kind of behavior policies. Authors might find [1] an interesting read. Adroit results in Table 1 kind of point towards something similar (BC and CQL are not that different in performance).

[1] Vladislav Kurenkov, Sergey Kolesnikov. "Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matters". AAAI 2022.

summary_of_the_review: While some claims about offline RL doing better than BC as "surprising" are not necessarily something I agree with, overall it's an interesting paper and allows the community to make progress towards better understanding of the tradeoffs between different policy learning mechanisms and would recommend acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper studies the sufficient conditions that VI-LCB would have a better worst-case sub-optimality gap than the gap of a BC algorithm, under expert data and noisy data. Moreover, the paper provide an empirical study to validate the theoretical results. 
main_review: Strengths
1. The paper studies an interesting problem: when an offline RL algorithm has advantages over a simple BC algorithm. I think the problem is in the interests of many RL practitioners. 
2. The paper provides a good overview on related works.  

Weaknesses (and questions)
1.	The paper proposes several sufficient conditions that VI-LCB could have a better worst-case sub-optimality gap. However, it does not provide practitioners a clear guidance on when they should run BC and when they should run VI-LCB, which I think is one of the main purpose of the paper. How should we check if the sufficient conditions are satisfied in real world problems? Some of the parameters are even unknown (e.g., b in Corollary 4.2). 
2.	I find it difficult to compare these upper bounds. Is it possible that some algorithms get larger bounds simply because the analysis is not tight?  Can we say that algorithm A is better than algorithm B because an upper bound (of the sup-optimality gap) for A is lower than an upper bound for B? 
3.	Some conditions are not well-justified. The paper studies some sufficient conditions that VI-LCB outperforms BC, however, some conditions seems a little arbitrary to me (e.g., condition 3.2, condition 4.1 and condition 4.2). I guess the paper has some specific applications in mind and makes these conditions based on these applications. However, this makes the results less clear and less general. For example, how do the results extend without condition 3.2? do we expect to just see an extra H term in all bounds without the condition? If the paper considers some specific applications, the paper should clearly mention it instead of saying the conclusion holds in general.   
4.	The paper studies a particular type of offline RL algorithm based on VI with LCB style penalty. However, the title and the paper seems to suggest that this particular algorithm can represent the class of offline RL algorithms? I don’t see how study one algorithm can generalize to a conclusion for offline RL algorithms? 
5.	I am not sure what is the novelty and significance of the theoretical results. It seems the theoretical results are mostly borrowed from existing literature with some modifications. I want to make sure I am not missing something important there. Can you explain more about the novelty or significance for the theoretical results? 

Minor questions and comments:
1. In the empirical section, does CQL lie in the VI-LCB framework? If not, why is it used in the empirical section while all theoretical results are for VI-LCB. 
2. VI-LCB with Bernstein style penalty is also used in [1], which is not cited in this paper. 

[1] Xie et al. (2021) “Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning”

summary_of_the_review: I am leaning towards a recommendation to reject the paper since I think the paper does not clearly answer the core question: whether should I run BC or offline RL for several reasons mentioned in the Main Review section. However, I am open to change my scores after author response. 

----
After the author response, most of my concerns were addressed. I think the paper provides good empirical and theoretical understandings of the comparison between BC and pessimistic VI, so I raised my score to a 6.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper provides an attempt to help practitioners answer the question "For what type of environments and datasets should we prefer Offline RL over Behavioural Cloning". To do so, the authors extend the previous work of [1] studying this problem for contextual bandits to MDPs. These theoretical results enable them to draw some conclusions on when we can expect Offline RL to have an edge on Behavioural cloning. To name a few:

- When the dataset provided is not optimal, offline RL can scale more favourable with the horizon especially for environments with horizon-independent returns or with a low volume of critical states,i.e. states for which there is no significant advantage to select one action over another.
- For long horizon tasks again, offline RL trained on noisy data displaying higher coverage can outperform BC trained on the same amount of expert demonstrations.

These theoretically grounded insights are then empirically validated on a tabular gridworld domain, after which they validate their findings on high dimensional offline RL problems (continuous control and navigation, and some atari games). On the tabular gridworld, the experiments strongly agree with their theoretical results. When turning to deep RL, the experiments mostly aligned although the various tuning mechanisms introduced in previous work to train efficiently offline RL algorithms introduce a few discrepancies.

1. Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, and Stuart Russell. Bridging offline reinforcement learning and imitation learning: A tale of pessimism. *arXiv preprint arXiv:2103.12021*,
main_review: ### Pros
- The quality and clarity of the writing are remarkable. The assumptions are clearly stated and transparent to the reader. The discussion around the theoretical results is well conducted and the insights are informative to the reader.
- The empirical evaluation on a tabular gridworld domain is appreciated as it removes the influence of function approximators on the results. The 3 types of girdworld instances provide the right settings to confront the theoretical results of section 4.
    - On a side note, I appreciate the care of using the suggestions introduced in [1] to report the empirical results of RL experiments.
    - I also appreciate the fact that the authors didn't shy away from highlighting potential discrepancies in their results due to the various tricks introduced in the literature to tune offline RL algorithms.

### Questions
- I would ask the authors to clarify the need for condition 3.2 as it seems that some of their results could be derived without it, e.g. theorem 4.1. How would the results be affected if we were to remove this condition?

### Minor Comments
- The space taken by the discussion around the theoretical results doesn't leave much space for the analysis of the experiments. As a reader, it is hard to digest and understand whether the results fully align with the theory.
- Add an explanation of the terms for theorem 4.2, like it is done for theorem 4.1.
1. Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G Bellemare. Deep reinforcement learning at the edge of the statistical precipice. *Advances in Neural Information Processing Systems*, 2021b.
summary_of_the_review: Overall, I consider this paper to be a great contribution to the research community. The question investigated is important for the research in offline RL and practitioners. The theoretical analysis is well conducted and meaningful insights can be drawn from it. The experimental results are adequately reported and support the findings.
review_title: null

Meta-review: This paper considers helping to decide whether behavior cloning or offline RL is likely to be more effective given a particular offline dataset. The reviewers initially appreciated the importance of insights into this question around how to best leverage an existing dataset. They also had some initial concerns, due in part because the theory is restricted to tabular settings, whereas many challenges typically arise when function approximators are used, the realisticness of the assumptions over the data collection process, and a number of places where further details or clarifications would better situate and strengthen the work. The authors gave very extensive responses to the feedback which made reviewers feel much more confident about the revised paper and resulted in significantly higher scores. Though there remains many interesting areas for future work, this paper makes an interesting contribution that may be of interest to many using batch decision making data.
Decision: Accept (Poster)