Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to use composition operators (product, mixture, negation) to modify the distribution for data generation. To do so, the authors proposed an energy-based parameterization of diffusion models. HMC (Hamiltonian Monte Carlo) is used for sampling rather than the reverse diffusion method. A simple illustrative and quantitative evaluation of the downstream tasks supports this design choice. In the experiment, the method's strength was demonstrated using many image generation tasks.  
strength_and_weaknesses: Strength:
- The paper clearly illustrated the challenge of composing data distributions for image generation and proposed an effective way to solve it.
- The idea is clearly described and the intuition is clear.
- Sufficient experiments to support its effectiveness.

Weakness:
- It might be better to discuss some failure cases for using this approach.
clarity,_quality,_novelty_and_reproducibility: It is a pleasure to read this paper. It is well-written, and the idea is very clearly illustrated and easy to follow. 
The idea is simple, powerful, and effective as illustrated by the paper.
Components of the method exist in the community, but using these techniques together to solve the challenge of composing distributions for image generation is novel.
summary_of_the_review: This paper proposed a simple and effective way to compose distributions for image generation. It is strong in both theory and experiment. The paper is nicely written and easy to follow.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This papers shows how diffusion models can be combined together to compose several relationships. The authors take an energy minimization perspective of diffusion models. From this perspective, sampling from the distribution can be seen as running an MCMC. Then, the authors show how different distributions can be composed together. They consider three types of distributions - product, mixture and negation. To sample from these distributions, the authors propose annhealed MCMC algorithm, which is explained in appendix. The authors then show some experimental results where their proposed sampler improves composition compared to baselines.
strength_and_weaknesses: Strengths:
The idea of composing distributions is an important one. One of the fundamental limitations of the current text to image models is the inability to compose several relationships, hence this problem is an important one to address. The paper takes a principled approach for this problem by considering an energy minimization perspective. They come up with good sampling algorithms that seem to improve over baseline.

Weakness:
One of the main weaknesses I see is the similarity between this paper and [1]. This paper addresses the exact same problem and they take an energy minimization perspective. This paper considers both product distributions and negations.  They use MCMC as well. The authors in this paper cite this work, but they don't mention the similarities and differences between this work. I find this very concerning. I think the difference lies in the samplers used. This paper uses HMC which [1] doesn't. But I'm still unable to assess how this is better than [1].

With respect to experimental validation, the authors show that using HMC improves over other baseline samplers. But, it would have been nice to include comparison with other papers like what [1] did. In fact, [1] had a table for CLEVR dataset. It would have been nice if the authors showed the comparison with that.

[1] Liu et al., "Compositional Visual Generation with Composable Diffusion Models", ECCV 2022
clarity,_quality,_novelty_and_reproducibility: Clarity: The authors can improve the writing a bit. I found it a bit hard to parse through the paper.

Quality and novelty: The paper seems to address an important problem. But I am a bit concerned about the similarity of this work with [1] as I mentioned above. The authors could have invested a bit more time in making experiments more rigorous. For instance, comparing with existing papers, other open source methods like GLIDE, stable diffusion, etc. 

Reproducibility: The authors mentioned many of the experimental details in the appendix.
summary_of_the_review: While the paper addresses an important problem, I feel experimental rigor is lacking and the authors could have done a better job comparing with other papers.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes methods to compose diffusion models as products, mixtures, and negations of distributions. The use of LMC and HMC samplers is first motivated by the analysis of the intermediate distributions given by the reverse diffusion. An energy-based parameterisation of diffusion models is presented which enables mixture composition and allows Metropolis corrections in the samplers. Empirical evaluation of compositional generation is provided on multiple datasets and tasks.

LMC: Langevin Monte Carlo,
HMC: Hamiltonian Monte Carlo
strength_and_weaknesses: [Strengths]
* The paper presents an interesting study on the use of different samplers and how they significantly change the quality compositional generation. 
* The proposal of the energy-based parameterization, although computationally expensive, is well motivated.
* Multiple qualitative and quantitative experiments have been presented that show the improvements offered by the proposed method.
* I appreciate that the authors have clearly discussed the limitations of their proposals. 
* The paper is overall well written but there exist some clarity issues (see weaknesses).

[Weaknesses]
* Clarity:
    * For readers unfamiliar with the background, please clearly define what is meant by reverse diffusion.
    * The jump to "improving sampling with MCMC" is abrupt. It is unclear how annealed MCMC improves the results over reverse diffusion. Furthermore, the authors have completely abstracted the details (e.g., the kernel k_t) in this section which forms one of the central theses of this work. I would like to see a more detailed discussion clearly explaining how LMC/HMC helps with compositional generation. 
* With the choice of LMC/HMC samplers, I would have liked to see a more detailed discussion on continuous-time diffusion models (e.g., [1]).
* I would have liked to see more examples/experiments of mixture and negation compositions.

[1] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020
clarity,_quality,_novelty_and_reproducibility: * The paper is well-written in general with some clarity issues that can be addressed.
* The overall quality of this work is good. It presents a interesting frameworks for compositional generation and empirical insights that result in improved performance.
* This work mostly builds on previously explored techniques but brings new insights and improved results.
* Experimental details have been provided in the appendix which aid reproducibility but a code release would have been better.
summary_of_the_review: Overall, the paper presents an interesting framework for compositional generation. The technical contribution is limited but the paper brings empirical insights that may be relevant to the generative modeling community. There also exist some issues with the clarity which can hopefully be addressed in a revision.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper aims to compose energy-based diffusion models for compositional generation. The authors first point out the deviation between sampling from the composed score function and sampling from the composed diffusion model. To this end, MCMC samplers are adopted in the sampling procedure to improve the composition generation of diffusion models, and with proper energy-based parameterizations, Metropolis corrections may be incorporated into the sampling process. The experimental results demonstrate that the proposed method realizes the composition of diffusion models in different ways and on different data.
strength_and_weaknesses: Strengths:
1. This paper has a good motivation. For diffusion models, compositionality can be an important property for many applications. For example, in the experimental results, the proposed method has shown good performance in the compositional scene generation and text-to-image generation.
2. This paper clearly explains the problems of diffusion models in compositional generation, and uses MCMC samplers which may incorporate Metropolis corrections by energy-based parameterized diffusion models. The effectiveness of the sampling procedure is validated in the experiments.

Weaknesses:
1. The novelty of the paper seems limited. All the building blocks of the proposed method are referred to existing works, including the composition operations in Section 3, MCMC samplers in Section 4.1 and energy-based parameterization in 4.2. 
2. The organization of the paper can be improved. Preliminary knowledge (including introduction to diffusion models in Section 2 and introduction to composition operations in Section 3) span more than three pages. In contrast, the main technical part is too short to elaborate the method.   
3. Although the paper points out the deviation between sampling from the composed score function and sampling from the composed diffusion model; however, this gap is not demonstrated on all tasks/datasets in the experiments.
4. In the abstract, it says “explore alternative ways to condition, modify, and reuse diffusion models” and “we investigate why certain types of composition fail using current techniques and present a number of solutions”; however it seems that there are no corresponding content in the main body of the paper, do I miss something or is it overstated in the abstract?

Some comments:
- Figure 3 lacks some necessary explanations, for example, what do the boxes with different colors indicate? 
- In Table 2 and Table 3, are EBM and Energy the same model?
- Some good experimental results of text-to-image are shown in Figure 5. Some failure cases of this task can also be shown and discussed.
clarity,_quality,_novelty_and_reproducibility: - The paper is easy to follow, but the organization can be improved.
- The novelty of the paper seems limited.
- The code is not provided.
summary_of_the_review: This paper is well motivated. It points out the deviation between sampling from the composed score function and sampling from the composed diffusion model. The novelty of the paper seems limited as all the building blocks of the proposed method are referred to existing works. The experimental results seem good.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper describes an energy-based diffusion models for compositional generation. The paper is well written and the ideas are clearly explained. The weakest part according to the reviews is the novelty, given similar approaches to composing diffusion models. This is a solid piece of work and might inform the community of the utility of these models. However, I feel that some of the points are not particularly well motivated. For example why should "AND" correspond exactly to taking the product of two distributions p1(x)p2(x) -- there are other ways to do this (for example p1(x)^alpha*p2(x)^beta). Is there a principle way to learn what the "best" way to combine distributions is?  On the whole this is a solid contribution, but the reviewer scores remained borderline after the discussion.
justification_for_why_not_higher_score: The reviewers remained largely unenthusiastic about this paper, despite it being technically good and well written. Fundamentally, I think it was considered too incremental.
Decision: Reject