Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes using a conditional diffusion policy for offline RL. The proposed method consists of a diffusion model as a behavior cloning term for policy regularization, and a standard policy improvement term via a Q function. The authors demonstrated the effectiveness of their method on a series of toy examples and the standard D4RL, AntMaze, Adroit, and the kitchen benchmarks.
strength_and_weaknesses: I liked this paper a lot, the authors presented a simple and straightforward method that yielded incredibly powerful results. Given recent advances in diffusion models, the idea of using a diffusion model for behavior cloning as a regularization term for offline RL seems like quite an obvious approach but is to the best of my knowledge quite novel. The paper itself is very well-written with a clear logical flow and was a joy to read. The authors also did a good job of placing their work in the context of other literature and comparing their work to other similar approaches. Experiments are very thorough and shows clear evidence of improvement compared to other baselines

I especially liked the toy example introduced in Section 5 and the Appendix and how the authors separated the behavior cloning part and the policy improvement part. The example gives clear visualization on how the proposed method is advantageous in dealing with multimodal data compared to other approaches. Though one minor suggestion I have is that I think the toy examples perhaps deserve a separate section in the main text.
clarity,_quality,_novelty_and_reproducibility: Paper was very clearly written, proposed method is novel to the best of my knowledge, and I do not see any major reproducibility issues.
summary_of_the_review: I think this work is well-presented and adds an important contribution to the community, I recommend its acceptance at this venue.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper presents Diffusion-QL, an offline RL method that can be seen as an actor-critic method, where the actor is a generative diffusion models, and the critic is a standard Q-function. The diffusion model is trained with supervised learning to imitate the data distribution. The main benefit of diffusion models is that they are able to accurately model the data distribution, even when it is multimodal, which is usually a challenge for previous methods. The Q-function is then used to guide the gradient during the training of the diffusion model towards actions with higher Q value. This combined approach prevents the agent from taking out-of-sample actions, while ensuring convergence towards the optimal actions among the explored regions of the state-action set. The authors introduce their approach as a policy regularisation method as it is constrained to a subset of the support of the data distribution.

First, the method is illustrated on a toy bandit problem showing how it is the only one able to accurately model the data distribution and the only one to reliably converge towards the optimal action among the baselines. Finally, further experiments show Diffusion-QL achieves competitive or state of the art results on the D4RL benchmark.
strength_and_weaknesses:  ### Strength
- The paper presents a fresh idea for offline-RL that fixes multiple issues from previous approaches.
- Identifying the main problem with previous policy regularisation approaches is the inaccurate approximation of the data distribution is insightful and leveraging diffusion models for directly modelling multimodal behaviour policies is an effective and sensible approach.
- Experimental results on the D4RL benchmark are consistent and very promising.
- The paper is well written and easy to read.

### Weaknesses
- Some important baselines are missing in the experiment sections that were designed to deal with multimodal data distributions, like the Behaviour Transformers (Shafiullah et al., 2022).
- The paper barely discusses the limitations of the proposed approach, like the high computational cost of sampling actions, which might prevent the approach from deployment in some real-world scenarios.

clarity,_quality,_novelty_and_reproducibility: **Clarity.** The paper is very easy to read, well organised, self-contained and with clear explanations.

**Quality.** The toy example supports the main hypothesis of the paper: diffusion models are able to model multimodal policies, which allows the Q-function to discover the in-sample highest value actions. The results in the D4RL benchmark are convincing that this is a promising approach. 

**Novelty.** On one side, the paper just combines diffusion models with a standard critic, and this is motivated from the realisation that the data distribution in offline RL can be rich and multimodal, which has already been discussed in previous works. On the other hand, the paper combines these two ideas in a simple, elegant, and effective manner. The result is a novel model for offline RL and will likely inspire further research, for example using other architectures beyond MLPs, condition on state-action sequences for tackling partial observability, replacing the Q-function with advantage estimates to guide the gradient, other noisy schemes, reducing the sampling cost, etc.

**Reproducibility.** The authors describe their architecture and hyperparameters in detail - I think they missed to give details about the optimizer though. But they have provided their code, which I expect (I haven't actually tried) will make the experiments easily and fully reproducible.
summary_of_the_review: This is an interesting paper that tackles an important and difficult problem, offline-RL, in a novel, elegant and effective manner. I believe the paper presents a proof of concept that can be extended in several ways and will likely inspire further research.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a new offline RL algorithm where the distribution of actions (conditionally to the state) is represented by a diffusion model. At the same time, existing methods usually use a simple gaussian. The approach is then quite simple: the diffusion can be used with a classical behavioral cloning loss (diffusion BC), and coupled with a Q function when reward is available (Diffusion QL). When associated with a Q function, the authors use a strategy that is similar to the TD3+BC method proposed in previous work. The method is then evaluated both on a toy bandit environment to better understand the differences between this model and the existing ones, and also on standard offline-RL benchmarks (D4RL). The diffusion algorithm is obtaining very good results and is very competitive w.r.t state-of-the-art.
strength_and_weaknesses: The paper is well-written and easy to follow. The proposed model is (very) simple -- which is a good point in my mind -- and I am sure that this paper is bringing interesting information to the community. If the idea is almost obvious (using the best-known and trendy generative model instead of simple gaussian models), I think that such a contribution provides important insights and researchers will be able to build on it. What the paper suggests is that the way we are used to capturing the distribution of actions (gaussian conditioned on the state) is not the right one since it is not able to capture multiple modes (while datasets, and particularly medium ones in D4RL are multi-modal since they have been built by using different acquisition policies). On that point, it seems particularly true for BC approaches where two experts may have taken two different actions in a similar state (and the average action would not be an expert action). What is less clear to me is that this also holds when the reward is available (particularly dense reward) where we can expect one action to 'dominate' all the other actions. For instance, in the toy problem (Figure 1), I don't understand why TD3+BC is not able to capture the same distribution as the one captured by Diffusion-QL, even using a simple gaussian. I would be interested in having more explanations on this point.

One question I also have is what would happen when using classical state-of-the-art methods with a mixture of Gaussians instead of a simple gaussian. Indeed, all the comparisons are made with a simple one-mode gaussian, and reimplementing a BC for instance with a mixture of 2 or 3 gaussians would give insight into the need to use diffusion models while simple approaches could work better. I would be very happy to see this kind of experiment in the ablation study for instance, but also in the toy environment which seems to be solvable through a mixture of Gaussians. Author could also compare their approach to 'implicit behavioral cloning' which proposes a solution to the problem of capturing multiple modes in BC.  

Minor remark: figures are too small, and axis are not readable. 
clarity,_quality,_novelty_and_reproducibility: * Well written
* Good execution
* Novel enough
* Good contribution
summary_of_the_review: This is a paper that is solid on both the idea and the execution of the idea. I do not see clear reasons to refuse such a paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to use a (conditional) diffusion model to represent a policy for the RL agent in the offline RL setting. The objective (3) consists of a Q-learning term and a behavior regularization term. Synthetic experiments on a bandit problem show the advantages of using diffusion models over other generative models, and experiments on common offline RL benchmarks show that the proposed diffusion-QL method can outperform alternative methods when tuned right.
strength_and_weaknesses: Strengths
- Writing is clear
- Good empirical performance

Weakness
- Limited novelty: the better performance mainly comes from a different parametrization of the policy borrowed from the generative model literature. It is a straightforward application.
- Some of the comparisons in the synthetic experiment are questionable. It is surprising to see that CVAE and MMD methods compared in Sec.5 fail so miserably for such a simple task. It is very likely that these methods are not tuned properly for the bandit problem (e.g., it is not clear why CVAE produces a diamond shape policy) No enough implementation detail is provided here so it is challenging to see what may have been done incorrectly.
- Some of the experiment details are missing and require further clarification. (a) The trade-off hyper-parameter is chosen on a dataset-to-dataset basis, which makes the applicability questionable. It is mentioned in Sec.6.1 and the appendix that eta (also see the discussion after Eq.(3)) is chosen differently for different benchmark problems. The actual values of eta are also not provided in the paper. (b) Page 7 mentioned that L_d is a “lagging indicator” and 2nd or 3rd checkpoints are chosen. This is not convincing at all and additional explanation is needed. As a result, the better performance becomes less convincing since it can be due to over-tuning.
clarity,_quality,_novelty_and_reproducibility: Clarity is good, but novelty is limited and reproducibility is questionable. (See weaknesses above)
summary_of_the_review: The paper proposes to use a diffusion model to parametrize the policy and claim that it can achieve better performance in the offline RL setting. However, the application is straightforward. It does not really show why such models are suitable for the task (the synthetic experiments are questionable) and important experiment details are missing. Hence the score.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes the use of diffusion policies for offline-RL. It is extremely straightforward in that it proposes to use diffusion policies trained to balance behavior cloning of the policy under which the data was collected, and a policy improvement derived by backpropogating through Q-value function. The Q-value function is learned in a standard way. There evaluation shows improvement on a variety of RL tasks compared to a variety of baselines. There's a discussion on hyperparameters and an ablation study on a few tasks. Though this paper is very straightforward, the combination deserves study. 

All reviewers were positive after the author response. The negatives that remains were some residual surprise on the poor performance of other generative models and the need to do dataset specific hyperparameter tuning, though this is discussed to some extent in the paper.
justification_for_why_not_higher_score: It is extremely straightforward in that it proposes to use diffusion policies to replace other generative models in offline-RL.
justification_for_why_not_lower_score: Though it is simple, the evaluation has merit and may encourage further work.
Decision: Accept: poster