Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a new pre-training method called Behavior Prior Representation (BPR) for offline RL that applies Behavioral Cloning with a decoder that learns a compact state representation that is then used by offline RL methods. This compact representation is shown to alleviate the implicit under-parameterization phenomenon that causes over-aliasing of features. The paper presents theoretical performance guarantees and demonstrates significant performance gains with BPR on the D4RL Offline RL benchmarks.
strength_and_weaknesses: Strengths:
- Clear motivation with plenty of references giving the unfamiliar reader the necessary context
- A large number of experiments showing significant performance gain
- Ablation studies to investigate Robustness to Visual Distractions and Effective Dimension
- Substantial effort to provide theoretical guarantees

Weaknesses:
- A few parts of the paper seem rushed (in contrast to the remaining part of the paper). E.g. the wordings in the Reproducibility Statement, "?other representation objectives?", and it seems to me that state and action are swapped in Figure 1 under BPR.
clarity,_quality,_novelty_and_reproducibility: The quality of the work is high given the amount, clarity, and significance of the experimental results. I do not qualify to give a proper evaluation of the entirety of the theoretical work. 
summary_of_the_review: This paper is of high quality, has a clear motivation, and a simple and well-founded approach that is evaluated with a substantial amount of experimental and theoretical work.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The work focuses on learning representation for offline reinforcement learning. The authors propose a representation learning method based on cloning the behaviour under which the offline dataset was collected. The main motivations of the work is to (1) reduce the difficulty of learning a policy from limited high-dimensional data by projecting the observations into a representative low-dimensional space, and (2) combat the under-parameterization problem of value networks that arises when bootstrapping is used by disentangling the representation learning from policy/value learning. 

The proposed representation learning method is independent of the policy learning RL algorithm and can thus be used with any offline RL algorithm. Authors provide theoretical policy improvement guarantees under some assumption and present empirical results, showing outperformance against baselines. 
strength_and_weaknesses: Strengths:

* The work addresses an important open problem in offline RL. 
* Very well motivated 
* The presentation is clear and coherent making it a pleasant read
* Some interesting theoretical guarantees are provided
* The proposed method is evaluated on different criteria (performance against baseline algorithms without pretrained representations, comparison against different representation objectives, change in the effective dimension of the data, and robustness to distractors in pixel-based inputs)

Weaknesses:

* The theoretical analysis provides some intuition on the kind of performance gain is expected when the representation is trained on a data under a policy $\pi_{\beta}$. This analysis essentially means that the quality of the $\pi_{BPR}$ highly depends on the behaviour policy. Authors should discuss address what to do in cases where the behavioural policy is highly adversarial for instance. What does that mean for the learned representation? Is it still beneficial to learn a BPR, or is it harmful then?

* When assessing the effectiveness of BPR against baselines (non-pretrained representation), a fairer comparison would account for the additional 100k training steps BPR used for pretraining. I understand the argument that the representation needed not to be retrained for all of the different offline RL methods, still it feels like an unfair comparison. 

* When comparing against other against other representation objectives, a couple things need clarifications:
  * How were the baseline selected? (the performance criterion used is from Schwarzer et al. 2021, however the SGI representation is not included in the baselines?)
  * Are the encoders for the different objectives of similar architecture/capacity?

* I would have like to see a discussion or some results comparing BRP with representations that embed the transition dynamics (e.g. self predictive representations) 
clarity,_quality,_novelty_and_reproducibility: Clarity: I found the paper clear and well structured

Quality: High quality, the clearly a lot of work went into this paper. The theoretical analysis in particular appears to be sound.

Novelty: I am not well versed enough in offline representation learning for RL to assess the novelty. Though I would like to bring to the authors attention the work of Le Lan et al. 2022 which also defines a notion of effective dimension in the context of representation learning. 

Reproducibility: I have not gone through the supplementary material in enough depth to comment on reproducibility. The information given in the main text would not be enough to reproduce the results. 


[1] Le Lan et al., "On the Generalization of Representations in Reinforcement Learning" 2022
summary_of_the_review: Interesting, well motivated and well presented paper. However the experiments are is not detailed enough/choices made are not justified clearly enough. No intuition is given for addressing the limitation of the work as it stands. For these reasons my score is 5.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper lies along the line of work exploring how to combine BC and offline RL. It takes the perspective of using BC as a state representation learning method, proposing to train an intermediate representation in the offline RL agent via BC (either as pre-training or co-training). Experimental results show that the proposed method outperforms other state representation learning methods, and offline RL from scratch, on the visual D4RL benchmark in “expert’ and “medium expert” modes.

Contributions:
Proposing BC as a state representation learning algorithm for offline RL
Theoretical analysis from a representation learning perspective
Empirical comparison to other state representation learning algorithms with visual inputs
strength_and_weaknesses: Questions and Comments

1) I am struggling to square Theorem 2 in this work with Theorem 3.4 in Li et al. 2006. If the data collection policy Beta is the optimal policy, then does a representation that satisfies Assumption 1.3 correspond to one that is \pi*-irrelevant in Li et al? I would appreciate if the authors could explain if and how the assumptions of Theorem 3.4 of Li et al. differ from their setting. The general question I think is one of representability versus learnability - it is possible to represent something but not be able to learn it. 

2) It seems that there may be an implicit dependence on the characteristics of the behavior policy that is not stated. Assuming full state support does not seem to be enough - the behavior policy could visit every state but never take a good action. At the limit, suppose the behavior policy always takes the same action - then BC can achieve perfectly accuracy by completely ignoring the state input. Don’t we therefore need an assumption on the minimum return of the behavior policy? Perhaps I missed it? Theorem 5 bounds the error between the estimated and true behavior policy, but this doesn’t address the question of the worst-case error of the policy learned via offline RL with the learned state representation. In the experiments, what happens if you run the algorithm on the “random” data collection setting of D4RL? 

3) I would recommend moving Figure 10 from Appendix G.5 to the main paper, as these are the main results - comparisons with other state representation learning methods on image observation inputs. What is the offline RL algorithm that is used for this experiment? What is the result here for the “random” data collection setting?

4) How is co-training with BPR different from TD3+BC? Is the difference that the BC loss is predicted with a separate head instead of being added to the TD3 loss?

5) Did you consider a probabilistic formulation for the state representation? If Z was treated as a latent random variable and the model optimized via variational inference, this would correspond to an information bottleneck on the representation Z, which would compress the information in the state representation.  

Nit
 - The theorem numbering starts at 2 instead of 1
clarity,_quality,_novelty_and_reproducibility: Clarity - The method and experiments are clearly explained; however, there may be a lack of clarity around the dependence on characteristics of the behavior policy.
Quality - Theoretical and experimental results are medium quality — see specific questions and comments above.
Novelty - The novelty of the method is medium. BC and offline RL have been combined in previous works (e.g., TD3+BC); however, this work proposes using BC as a state representation learning method and analyzes it from this perspective.
Reproducibility - Code is provided and datasets are public, reproducibility is high.

summary_of_the_review: I am concerned about the theoretical results regarding learnability of the optimal policy using this representation, and particularly how this might depend on the performance of the behavior policy. I would like to see this tested empirically as well by evaluating on the “random” data collection setting in D4RL. It is still a valuable contribution if the method only works well with certain data collection policies - however, this should be explained and tested empirically. 

I am currently unconvinced about the correctness of the claims without assumptions on the behavior policy (see Questions above). This is reflected in my current scores, which I will revise upward if my concerns are addressed. 


------ Update 12/9/22 --------

Thanks to the authors for the response clarifying the theoretical contributions. I also appreciate the changes the authors made to the paper to state explicitly the dependence of the quality of the learned state representation on the performance of the behavior policy, and to add experiments demonstrating this dependence. 

The results in this submission are about the relative performance of the learned policy to the behavior policy, and so the sufficiency of the learned representations for representing the optimal policy (as explored in Li et al. ) is a moot point (as the authors point out). 

As a side note, I do not understand the authors’ explanation: “Instead, we relax their strong requirements by allowing the value estimation in the ground MDP.” How is value estimation being done in the ground MDP when you run the offline RL algorithm in the “abstract” MDP defined by the learned state representation. In the updated appendix of the paper (page 18), it says that in contrast to Li et al., you “only change the state space,” leaving transition and reward functions unchanged. How can this be? 

My concerns for the theoretical section of the paper have been addressed and I have adjusted my score for the paper accordingly. 

However, I continue to think the limitation of the method to high-performing behavior policies is a significant limitation in the context of offline RL, which is often predicated on the existence of suboptimal data (indeed, when expert data is available, BC methods have been shown to be quite competitive with offline RL). In the "random" setting D4RL experiments shown in Figure 13, BPR is the worst-performing method. I also think the novelty algorithmically is rather low, consisting of a few tweaks from representation learning algorithms applied to BC. Therefore most of the contribution of the paper is carried in the empirical results that the method works better than other forms of combining BC and offline RL. I think this is a valuable enough contribution and timely given the strong interest in this topic, so I cautiously recommend acceptance. 

I encourage the authors to further clarify the wording in the theoretical sections to impart the conclusions more clearly, and also to de-clutter the appendix (e.g., by removing pseudo-code) to make it easier to find results (such as the "random" D4RL experiment illustrating a failure case of the method).
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The idea behind this contribution can be summed up easily (which is a strength): pre-train using behavioral cloning, and then use the learned representations to do offline RL. The paper shows both theoretically and empirically that this works well and gives appreciable benefits to standard offline RL methods. I and the reviewers concur that this is good work with some general applicability.
justification_for_why_not_higher_score: While this is a clearly a good paper, the idea is not novel enough (imho) for a spotlight presentation.
justification_for_why_not_lower_score: The paper is well-written, the results are good, and there is sufficient novelty to accept the paper.
Decision: Accept: poster