Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents Decision Diffuser, a conditional generative model for sequential decision making. It frames offline sequential decision making as conditional generative modeling by considering  two other variables: constraints and skills. Conditioning on a single constraint or skill during training leads to behaviors at test-time that can satisfy several constraints together or demonstrates a composition of skills. Experiments are conducted on a couple of different decision making tasks.
strength_and_weaknesses: Pros:
This paper is well motivated and easy to follow. Inspired from diffusion models in vision domain, this paper formulate the decision making process as a condition generation problem and which can naturally achieved by leveraging diffusion models. I like the way that this paper very clearly introducing the technical background and formulate the problem. From the experimental results, the proposed Decision Diffuser is promising on a couple of evaluating tasks.

Cos:
My main concern is the limited technical novelty. The key contribution of this work is to formulate the decision making problems as a conditional generation problem. Based on this, this paper train a diffusion model on offline datasets. However, it is more like an application of diffusion models on decision making tasks. I don't see obvious novelty from either model design and/or training objectives.
The evaluations are weak. It only evaluate the DD variants on a few tasks. No comparisons of DD with the other state-of-the-art approaches are shown. Also, no ablation and discussion are shown. It is not convincing to me without such extensive study.
clarity,_quality,_novelty_and_reproducibility: This paper is well motivated and easy to follow. Inspired from diffusion models in vision domain, this paper formulate the decision making process as a condition generation problem and which can naturally achieved by leveraging diffusion models. I like the way that this paper very clearly introducing the technical background and formulate the problem. From the experimental results, the proposed Decision Diffuser is promising on a couple of evaluating tasks.
My main concern is the limited technical novelty. The key contribution of this work is to formulate the decision making problems as a conditional generation problem. Based on this, this paper train a diffusion model on offline datasets. However, it is more like an application of diffusion models on decision making tasks. I don't see obvious novelty from either model design and/or training objectives.
The evaluations are weak. It only evaluate the DD variants on a few tasks. No comparisons of DD with the other state-of-the-art approaches are shown. Also, no ablation and discussion are shown. It is not convincing to me without such extensive study.
It did not submit the source code, so the reproducibility is hard to be validated.
summary_of_the_review: This paper is well motivated and easy to follow. Inspired from diffusion models in vision domain, this paper formulate the decision making process as a condition generation problem and which can naturally achieved by leveraging diffusion models. I like the way that this paper very clearly introducing the technical background and formulate the problem. From the experimental results, the proposed Decision Diffuser is promising on a couple of evaluating tasks.
My main concern is the limited technical novelty. The key contribution of this work is to formulate the decision making problems as a conditional generation problem. Based on this, this paper train a diffusion model on offline datasets. However, it is more like an application of diffusion models on decision making tasks. I don't see obvious novelty from either model design and/or training objectives.
The evaluations are weak. It only evaluate the DD variants on a few tasks. No comparisons of DD with the other state-of-the-art approaches are shown. Also, no ablation and discussion are shown. It is not convincing to me without such extensive study.
In addition, it did not submit the source code, so the reproducibility is hard to be validated.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: **Update after rebuttal** I am very happy to see a very extensive and detailed authors' rebuttal with a number of additional evaluations and results. Most of my questions have been answered and my criticism has been addressed to a sufficient degree. Though there could be some detailed follow-up discussions, I think overall the paper is now ready for acceptance - I am on the fence between a 6 and an 8, but am very slightly leaning towards the latter, so I will raise my score from 5 to 8.

The paper explores the use of a diffusion model to generate reward-conditioned state-trajectories to reach goals and satisfy constraints in MDPs given a large set of state-action-reward trajectories (solving decision-making via conditional generative modeling). In contrast to previous work on synthesizing high-expected-return policies from expert data, this paper uses a classifier-free guided diffusion model. Additionally, the paper proposes to diffuse over state-trajectories only, rather than state-reward trajectories; which is empirically shown to lead better policies but requires learning an inverse dynamics model. The method is compared against a number of state-of-the-art competitors on two standard benchmarks (D4RL tasks, and Kuka Block stacking tasks). Finally, the paper shows how conditional generation of trajectories can not only be used for conditioning on high-rewards but also other subsets of training trajectories which either satisfy certain constraints or belong to a certain type of skill (such as a particular gait for running).
strength_and_weaknesses: **Main contributions, Impact**
1) Use of classifier-free guided diffusion. Classifier-guided diffusion requires a Q-value estimation procedure such that Q-values can be used to guide diffusion (which has been explored in Janner et al. 2022). The current work avoids this via classifier-free guidance which requires low-temperature sampling in datasets where demonstrations are of mixed quality. Empirically, this leads to improvements over previous methods - with a grain of salt it seems to be easier to pick out the high performing trajectories than learning good Q-value estimates. Impact: low to medium - it remains unclear under what exact conditions (both theoretically and empirically) classifier-free guidance / low-temperature sampling outperforms Q-value guidance and why.

2) Diffusion of state-trajectories only, rather than state-action trajectories (which is compared to in the paper and shown to perform worse). This comes at the cost of requiring a good inverse dynamics model. Current impact: low - similar to 1) the empirical results are in favor of the version proposed in the paper but it remains unclear what the practical cost of requiring the inverse dynamics model is; in particular how robust the method is against errors and imperfections in the inverse dynamics estimate (is performance very brittle or quite robust). Under what conditions is it better to rely on an approximate inverse dynamics model, and when is it better to diffuse state-action trajectories?

3) The paper shows how to incorporate combinations of constraints and skills: by identifying subsets of training trajectories corresponding to an individual constraint/skill and augmenting them with a label that can later be used to condition on. The approach is fairly straightforward, but interesting. Impact: low - the approach is currently only minimally explored in the paper and further conceptual/theoretical characterization of the kinds of constraints and the corresponding requirements for the dataset would be needed for higher impact.

**Strengths**
 *  Very well written paper
 *  Empirical results show benefits of proposed method compared to high-quality SOTA methods on standard benchmarks
 * Incorporating constraints and combining skills via constraints is interesting.

**Weaknesses**
 *  Often the paper’s main aim seems to be to reach the minimal result sufficient for publication, which is mainly to achieve good benchmark results. While this is a valid strategy to avoid risks, it also creates a sense of lack of ambition and depth. I personally think there are some very interesting ideas presented in the paper but they often seems to be addressed in a minimal sense. Beyond the empirical comparisons, a strong characterization of the advantages and disadvantages of these ideas is missing; and I would be very excited to see such a discussion and analysis (both empirically and theoretically). I think it would help raise the significance of each of the main contributions mentioned above.
 * To make the point above more concrete: 
   * It remains unclear when precisely and why classifier-free guidance and state-prediction only is preferable - when and why is it beneficial to avoid estimating Q-values at the cost of requiring an inverse dynamics model (how robust is each approach to errors in the approximate Q-values/dynamics, what are the conditions w.r.t. the dataset that make low-temperature sampling a good choice, etc).
   * It remains unclear what exact conditions are needed for constraints/skills to be “combinable” (see comment under improvements).
 * The current experiments on combining skills are a bit hard to interpret - while they lead to somewhat visually different gaits it is unclear whether these should be considered successful combinations of skills. What’s missing is a clearly stated and (quantitatively) measurable goal for combining skills (which is well defined in the case of combining constraints).

**Improvements**
1) Naively it seems that the approach introduced allows to combine individual constraints / skills, as long as there is a separate subset of training trajectories for each constraint/skill and the combination at test time is a conjunction of constraints such that the intersection of the corresponding sets of trajectories constitutes a set of valid solutions (which corresponds to the situation in Fig 1). Is this a hard condition? Can something theoretical be said about the allowed combinations of individual constraints (conjunction, disjunction, exclusive disjunction)? What happens theoretically and empirically if satisfying a combination of constraints requires novel trajectories that have no support in the training data (i.e. such that simply “filtering” out the right training trajectories via conditioning and some mild generalization is not sufficient)?

2) Combining skills needs some quantitative goal/metric, and an empirical evaluation. The videos are interesting and promising, but it is unclear how to interpret the results, let alone how other methods could be compared in terms of performance.

3) Perturbation analysis comparing the Diffuser and Decision Diffuser in terms of robustness to noise or errors in the Q-value estimates/inverse-dynamics model respectively. While theoretical results would be nice, empirical results are probably easier to obtain and might already be insightful. Related: how well do both methods handle quite noisy dynamics? Since the latter two requires running additional experiments, potentially even designing toy environments, I do not have a hard expectation to see these results after the rebuttal; but they would certainly make the paper stronger.

4) Discussion of limitations should be expanded
 * What are restrictions/requirements on the dataset, particularly w.r.t. allowing for combinable constraints/skills?
 * The current method is limited to fully observable MDPs, partial observability and latent variables might pose a hard to overcome problem, with self-delusions [1] potentially being a fundamental obstacle).
 * It seems unlikely that the method can be used to improve significantly beyond the performance of demonstrations (unlike RL, at least in theory). 

**Minor comments**

A) What is the relation between constraint conditioning as presented in the paper and more general task conditioning (via natural language prompts) as in e.g. DeepMind’s Gato?

B) Sec. 3 “It is useful to solve RL from offline data, both without relying on TD-learning and without risking distribution-shift.” - what are the conditions/requirements for the latter to be possible (at least informally)? In order to not *risk distribution shift* strong conditions on training-data coverage seem required.

C) What is the fundamental difference between constraints and skills - are they more than simply two different names to refer to subsets of trajectories in the training data?

D) “In contrast to these works, in addition to modeling returns, Decision Diffuser can also model constraints or skills and generate novel behaviors by flexibly combining multiple constraints or skills during test time.” - while this has not been explored in previous publications, is there a fundamental problem that prevents previous methods from using the same approach as used in this paper (i.e. augmenting data with a one-hot indicator for constraints/skills, and conditioning on that during generation)?

E) P4, typo: “choices fof diffusion”

[1] Shaking the foundations: delusions in sequence models for interaction and control, Ortega et al. 2021

clarity,_quality,_novelty_and_reproducibility: The paper is very well written, perhaps a slightly more expanded discussion and detailed comparison against previously proposed methods such as the Diffuser could be helpful to clearly and compactly state the novelty. The quality of the experiments shown is high, including comparison against strong SOTA methods. What I would have liked to see for an even higher quality paper is some critical discussion and analysis (via ablations or control experiments) of the main innovations (see Improvements). The method is a fairly incremental improvement of e.g. the Diffuser and the idea of using generative modeling for decision making has by now been widely explored - yet I think the paper has sufficient novelty. Sufficient details for reproducibility of the work seem present in the appendix.
summary_of_the_review: The paper makes some incremental, but sensible improvements to previously proposed work - and the empirical results justify the modifications. While the main ingredients that went into the improvements have been reported before, the particular combination is novel. To me personally, the paper could be more impactful and of higher quality by justifying the introduced changes beyond mere overall performance on some benchmarks and discussing advantages and disadvantages in detail (including some insight into when and why the introduced modifications are beneficial compared to the previous method). Another novelty introduced is the possibility to combine constraints/skills - this is an interesting idea, but currently it is explored rather superficially, a deeper exploration (as suggested earlier in my review) would make the work stronger. Overall I am quite on the fence for this paper - while there is a clear and sensible main idea and empirical results are in favor of the idea, the paper also has a bit of a flavor of “we tried something and it worked, but we didn’t really look into it any further”. I think that’s potentially a missed chance, since the paper could be quite strong with a bit more work and attempting to answer some of the harder questions. I am therefore currently voting for a borderline rejection, and would not be upset if the paper got accepted - but to me personally the paper is currently on  the level of a very well written and very strong workshop contribution; promising main results but a bit of a lack in depth and clear and well-understood main findings. I am happy to reconsider my final verdict in light of the other reviews and author discussion.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes Decision Diffuser, a diffusion-based model for sequential decision making where only the states of a trajectory are modeled and an inverse dynamics model is used to predict actions. Classifier free guidance is used to bring in conditional information, in the form of maximizing returns, satisfying constraints or composing together skills. This is used with low-temperature sampling at inference time yielding improved results. The proposed method is evaluated to generate trajectories for return maximization with offline reinforcement learning datasets. Kuka Block stacking and Unitree-go-running environments are used to assess the performance of generated trajectories in constraint satisfaction and skill composition tasks, respectively.

strength_and_weaknesses: Strengths:
- The paper is well written. Diffusion modeling for planning is an exciting and timely contribution given that diffusion models are beating state of the art in various other tasks in synthesis.
- The proposed technique is shown to tackle compositionality of skills and constraints at inference time, which is interesting.

Weaknesses:
- One limitation of diffusion models is that the generations are slow. It would be great if authors can comment on the runtime characteristics of the proposed model.
- It would be good to discuss limitations of the model. Some questions I have: 
  * How are the out-of-distribution generalization characteristics of the model? 
  * Can the model capture variable length trajectories? 
  * I am intrigued by the skill composition idea. Currently only “AND” style composition is demonstrated. What other compositions does this method can support?
  * How does the method perform with limited data?

Other comments:
- The loss function used to train the model combines losses which are used to train the diffusion model and inverse dynamics models equally. What implications does equal weighting have?
- The paper relies on diffusion models and class conditioning capabilities of such models (i.e. classifier free guidance) but I found it odd that the modeling choice is not mentioned in the abstract. 
- What is the performance metric used in Figure 4? Similarly, Table 3 is lacking an explanation of the metric used. 

clarity,_quality,_novelty_and_reproducibility: The paper is well written and organized. The proposed method is evaluated empirically across a broad range of offline reinforcement learning scenarios for return maximization. Skill composition and constraint satisfaction are also explored.

Diffusion models are proposed to be used for sequential trajectory data in Jammer et al. (2022), where both states and actions are modeled jointly. The model proposed in this paper separates these two and uses an inverse dynamics model to model actions. Furthermore, they adopt a classifier free guidance approach (as opposed to classifier guidance in Jammer et al., 2022). These incremental advancements yield superior results in offline reinforcement learning settings.

Janner, M., Du, Y., Tenenbaum, J.B. and Levine, S., 2022. Planning with Diffusion for Flexible Behavior Synthesis. arXiv preprint arXiv:2205.09991.
summary_of_the_review: The paper explores diffusion models for planning, which is a timely contribution since diffusion models are beating state of the art in various problems. It offers incremental advancements over the state of the art in this domain however the proposed advancements are shown to yield superior performance on various tasks. I think this is a good paper and I recommend acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents the decision diffuser for decision-making. The core idea is to model the decision making as a conditional generative model, which is different from the complex reinforcement learning-based works. Specifically, the method models a policy as a conditional generative model where the diffusion process is applied over states and gets return-maximizing trajectories. The experimental results show the performance on par with recent offline RL-based methods. 
strength_and_weaknesses: ## Strength
- Modeling decision-making as the conditional generative model is novel and also interesting to the community.
- Via conditional generative process, the method shows the ability to compose skills in order to maximize the rewards. This is impressive. 
- The paper in general is of high quality in terms of supportive experiments, and motivation. 

## Weakness
I'm not an expert in decision-making/Reinforcement learning. So I'm not entirely sure if I follow the problem definition. To be specific, what's the design choice that leads to the ability to compose skills? And do other reinforcement learning-based method have the same ability? It would be better to make it simpler. 
clarity,_quality,_novelty_and_reproducibility: The paper is of high quality, clarity, and novelty. Since I'm not an expert in refinement learning/decision-making. I'm not entirely sure about the reproducibility.
summary_of_the_review: From my perspective, the paper is of high quality. It presents a well-motivated story. 
- Good motivation for modeling decision-making as a generative model. 
- More importantly, the method shows the ability to compose skills, which impresses me a lot.
- The experimental results are also impressive. Though the conditional generative model is simple, it shows the performance on par with other complex reinforcement learning. 

Although I'm not very clear about some details of the diffusion process in the composition of skills, I tend to accept this paper given the impressive results.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper introduces the Decision Diffuser (DD) model, a novel application of diffusion models to offline reinforcement learning. Key to their approach is the use of classifier-free guidance for conditioning on returns, skills or constraints, which allows for dynamic recombination of behaviors at test time and the incorporation of an inverse dynamics model for generating actions from generated state sequences.

Reviewers [9svn, pyhf, WRRm, AC] agree that the method is shown to be effective across a broad range of challenging environments, and baselines spanning classical offline RL methods, and other recent advances based on conditional generative modeling (Decision Transformer, Diffuser). Furthermore, the paper is clearly written and the revised manuscript has incorporated some notable improvements (clarifications, and many ablations) which address most if not all of the reviewers’ concern. Well done! In particular, I commend the authors’ for their clear and concise explanation of diffusion models and classifier-free guidance which will be most useful to the RL community.

As mentioned by [pyhf], this is an exciting and timely contribution which will be of great interest to ICLR: both to the general RL community, but also the generative models crowd as another successful application of diffusion models. For the RL crowd, not online does this paper offer a convincing alternative to offline RL, it also adds significant weight to the general question posed by the papers’ title. Perhaps conditional generative modeling is enough? Either way, I’m excited to see where this work leads us. Congratulations to the authors!

justification_for_why_not_lower_score: I think this paper will be of broad interest to the ICLR community (both on the RL and generative models side) and as such would be well suited to an oral presentation. The paper is clearly written, with many interesting ablations incorporated during the rebuttal phase, which means everyone will likely find something of interest and something to learn from the talk.

An oral presentation would also give exposure to the high-level question posed by the paper's title: is conditional generative modeling enough for RL? Regardless of the answer, these models are clearly powerful and the RL community should pay attention. An oral is the best way to do this, and hopefully leads to some of these advances being incorporated in more classical online approaches based on temporal difference learning.
Decision: Accept: notable-top-5%