Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work uses a diffusion model as an expressive generative model for behavior cloning. They then turn this into a offline RL algorithm with model improvement by learning a Q-function and performing a form of importance sampling to rejection sample action samples with high value by sampling from the behavior clone model and resampling based on Q. They test this on some standard offline RL benchmarks.

strength_and_weaknesses: Strengths:
1. The paper is well written and explains the method.

2. The method is well motivated.

3. The empirical results show good.

4. The topic is off significant interest.

Weaknesses:
1. I have concerns about the reproducibility, I believe the norm (particularly for this topic) is to include the source code in the submission rather than a promise of future code. Generally, particularly in RL, people have struggled to exactly reproduce results just from pseudocode.

For example, I did not see in Appendix C the number of diffusion steps used (quite an important detail). It would also be helpful to discuss the runtime costs of sampling from diffusion models.

Ideally the results would be over more than 3 seeds, but this is not vital.

Other:

[1] Is a closely related work that uses diffusion models for offline RL that should be mentioned in related work (although the method is not identical). It also makes similar claims about the importance of expressive, multi-modal policies.

[1] Wang, Zhendong, Jonathan J. Hunt, and Mingyuan Zhou. "Diffusion policies as an expressive policy class for offline reinforcement learning." arXiv preprint arXiv:2208.06193 (2022).

clarity,_quality,_novelty_and_reproducibility: Clarity: This paper is fairly well written.

Quality: The paper is of high quality.

Novelty: The paper applies diffusion models to offline RL and uses a well-known solution to solve for high value actions while minimizing KL divergence from the behavior policy. The only other work applying diffusion models to model-free RL is [1], which was presumably contemporaneous with this work and the specifics of the two approaches are distinct.

Reproducibility: The source code is not provided (it is suggested that it will be made available later). For this style of paper, with benchmarks on a standardized set of tasks in a rapidly area of research this seems below the norm. In particular, I would imagine people would struggle to reproduce these results without access to the code.

[Updated during rebuttal. The authors now provide source code. I did not test it but this addresses my concerns about reproducibility].
summary_of_the_review: A well-present piece of work applying diffusion modeling to offline RL. It would be strengthened by making the source code available to ensure it can be reproduced and built on [I note this concern is fixable during the rebuttal period].

[Updated]

The authors have uploaded source code, cited additional work and made other improvements. I have increase my rating.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to learn the offline RL policy by a coupling of pure behavior cloning and importance sampling. The behavior cloning is modeled by a denoising diffusion probabilistic model, and an implicit in-sample Q-learning is applied to estimate the importance sampling weight. Experiments are done in the D4RL benchmark datasets to demonstrate the effectiveness of the method. 
strength_and_weaknesses: Strengths: 
1. The performance of SfBC evaluated on D4RL is comparable to the current SOTA methods. 
2. The authors demonstrate the importance of using expressive policy class through toy examples. 
3. The paper is easy to follow and well written. 

Weaknesses:
1. This paper lacks novelty. Policy improvement via weighted regression has been well studied in many papers, such as IQL and AWAC paper. The proposed Q-learning via in-sample planning is motivated from Ma et al. 2022. Applying diffusion models in offline RL is new while using diffusion models for behavior cloning seems an easy plug-in.  
2. The paper misses some details about implementation. 
3. Computational complexity is a concern and not discussed. 

clarity,_quality,_novelty_and_reproducibility: The paper is well structured and easy to read. The originality is limited. Code is not provided and the paper misses some implementation details, so the reproducibility is limited. 
summary_of_the_review: This paper proposes to separate the policy learning into two parts: 1) pure behavior cloning via diffusion models; 2) importance sampling via in-sample Q-learning. The idea of using diffusion models as an expressive policy is appealing. However, I do have concerns about the novelty and computational complexity. 

As I mentioned in the weaknesses, most of the components from the paper are proposed by prior works. Moreover, by looking at the Algorithm 1, performing implicit in-sample planning has high computational complexity. There are two for loops, one for K and the other for N, which is dataset size and usually huge. Diffusion models are notoriously slow in sampling, while the training algorithm needs samples from diffusion models within the K$\times$N loop, which has too high complexity.
I also have some detailed questions as follows: 
1. What is the number of timesteps for the diffusion model used in the paper? I guess an ablation study on the number timesteps is needed to show the effect of it.  
2. Better provide a training curve in the appendix to show the training stability. 
3. For the reported values in Table 1, what are these values? Are them the statistics based on the last points after the training or the highest points during the training? 
5. The policy network is much larger than the current policy networks, which makes the computational cost expensive and comparison not very fair. 
6. Expect to an ablation study on the K, since it is a key parameter in Algorithm 1. 
7. Minor: typo in Equation (12), I think it is $R^{(k-1)}_{n+1}$.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper introduces a method for offline reinforcement learning that involves fitting a more expressive density model and an episodic planning technique. The method demonstrates decent results on D4RL, a standard benchmark for offline reinforcement learning.
strength_and_weaknesses: # Strength 
* The approach is simple and produces strong results on the standard benchmark.

# Weaknesses 
* Right now, the main weakness is a lack of clarity. It is not clear how the method is implemented, and the paper can be significantly improved by expanding the implementation details.
* The approach has a limited novelty. Expressive policies have been previously studied in EMAQ [1]. Moreover, the value computation technique is similar to a soft version of EMAQ.
* The related work section misses some important prior work in particular [1], which also introduces a similar approach.
* At the end of section 4.2, it's mentioned that the n-step operator proposed in this work works in the stochastic setting. However, this statement is not completely clear. For example, does it mean that it's unbiased? 


[1] EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL
Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, Shixiang Shane Gu
clarity,_quality,_novelty_and_reproducibility: The paper needs to be improve in terms of clarify. Some important prior work is missing. Given a limited discussion of implementation details, it might be difficult to reproduce the results in the paper.

Please see the strengths and weakness for more details
summary_of_the_review: In overall it's a simple a well-performing method. However, it has limited novelty and the writing needs to be improved before the paper can be accepted.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper is somewhat borderline, since the novelty of the method is somewhat limited. That said, the paper is generally well-executed with clear motivation and nice results on a number of challenging offline RL problems, including ant-maze and kitchen. The author response also helped address some reviewer concerns, i.e. around computational complexity and implementation details. Thus, I recommend accept.

There are a couple other related works that I think should be included, namely PLAS (https://arxiv.org/abs/2011.07213) and LAPO (https://openreview.net/forum?id=pHd0v8W30O). I'd encourage the authors to incorporate these into the related work discussion in the revised version.

justification_for_why_not_higher_score: The novelty is limited.
justification_for_why_not_lower_score: The paper is generally well-executed with clear motivation and nice results on a number of challenging offline RL problems, including ant-maze and kitchen. The author response also helped address some reviewer concerns, i.e. around computational complexity and implementation details.
Decision: Accept: poster