Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper under consideration proposes a method for inverting stochastic diffusion models. Based on this method the authors come up with an (conditional) unpaired image-to-image translation approach called “CycleDiffusion”. The validity of the proposed methodology is verified by several applications.
strength_and_weaknesses: Strength:
1) The proposed idea is clear and practically applicable. One can easily apply the proposed framework for a general stochastic diffusion model.

2) The experiments look quite convincing.

Weaknesses:
1) The idea of DPM-Encoder itself seems to be quite easy and straightforward. All of the papers dealing with stochastic diffusion models (like DDPM) actually mentioned the forward form of the stochastic dynamic (i.e. how to obtain latent code $x_T$ from the original image $x_0$) but didn’t stress the importance of the dynamic. Therefore, the only theoretical contribution of the paper under consideration is that the authors highlight the importance of forward stochastic dynamics (for the problem of unpaired image-to-image generation related applications)

2) As it even mentioned in the article itself, the methodology of stochastic diffusion based unpaired image-to-image generation is under question and requires theoretical verification.
clarity,_quality,_novelty_and_reproducibility: The paper is over-all well-written. The results are novel and original.

The authors propose some new ideas (usefulness of the forward dynamic in context of stochastic models) and explicitly state the hypothesis that the diffusion models with similar noise schedule share latent spaces. 

 The practical quality of the paper is good since the authors provide a broad comparison of their method with existing approaches. One question: How do ILVR, SDEdit, EGSDE models apply to the problem of unpaired image-to-image translation (Cat vs Dog and Wild vs Dog)? As I understand, the mentioned methods are not directly aimed at the unpaired im-to-im translation by design.

 The code is not provided in the supplementary materials.

summary_of_the_review: The authors provide a new method for solving (conditional) unpaired im-to-im problems based on stochastic diffusion models. In spite of the interesting methodological proposition and good practical results, the method requires further theoretical investigation.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a way of deriving an encoder from a pretrained diffusion model. In particular, this encoder is stochastic and invertible, which makes it different from existing attempts. The authors show that this encoder can be deployed in an image to image translation setting, and demonstrates interesting results. Moreover, a unified guidance method is proposed based on the derived latent representations. 
strength_and_weaknesses: 1. The idea of defining the stochastic encoder is simple but interesting. 
2. Being able to encode an image and using the code for image to image translation or editing is clever.
3. The empirical results on both image to image translation and text guided edits are impressive. 
clarity,_quality,_novelty_and_reproducibility: 1. While the proposed stochastic encoder is interesting, and I wonder how it compares with the ODE based deterministic encoder empirically. Will the sotchasticity give any benefits on either the results or computational cost over the deterministic version? 

2. I have found that the plug-and-play guidance part rather disjoint from the main contribution and not properly justified. My understanding of it is that the authors propose to formulate a text conditioned energy function using a CLIP model. One can then guide a given generative model by optimizing its latents, which applies to both GANs and diffusion models. My question is that how is this related to the stochastic encoder? Why can't you just optimize the initial noise $x_T$ following a DDIM inference procedure? Maybe I'm missing something but I think it'd be good to clarify on this. 

summary_of_the_review: Overall I think this is a promising paper with a simple idea and interesting results. I'd like to hear clarifications on my questions during the rebuttal phase.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper provides a unified setting among generative models. In particular it offers a perspective  on diffusion models (DM) that cast them in a framework more similar to GANs, which admit a Gaussian formulation of the latent space. This point of view is then leveraged in three settings. 
Contributions: 1. provide a formulation of DM with Gaussian latent code  (introducing the concept of DPM-encoder that maps images into a latent space). 2. Propose CycleDiffusion for a) image-to-image translation and b) to show that text-to-image diffusion models can be used as zero-shot image-to-image editors. 3. Rely on the proposed perspective which unifies Diffusion Models and GANs to perform plug-and-play guidance.   
strength_and_weaknesses: Section 3.2: do I understand correctly that your DPM-Encoder is a proposal for how to build a z such that x = G(z) for z ~ DPMEnc(z | x, G), which amount to just concatenating x_T with \eps_t as described?  

Can you clarify the following: in section 3.1 I understood that in your unified framework, z is drawn from a Gaussian. In equation (6), it doesn't seem to be the case. In section 3.2, is the Gaussian latent  abandoned? 
 
I am a bit confused about the proof in Proposition 1. Do you use induction ((show it’s true for 0, assume it’s true for t, show it’s true for t+1)
) or backward induction (show it’s true for T, assume it’s true for t, show it’s true for t-1)  ?
Also,  if the goal here to show that G(z) = x for any z sampled from DPMEnc ( | x), in the statement I would write x = G(z) =: \bar{x}, just for clarity. 

Section 3.3: 
Can you clarify what you mean with “our defined latent codes contain all randomness?”
“ we can formalize the above finding as a common latent space emerges..." :can you clarify this sentence (what does it mean pragmatically?) ?  
How is the concept of  ‘common’ defined and measured? 

What do you mean “D_1 and D_2 do not need to be two datasets?” 

Section  4.1 
Peak Signal-to-Noise Ratio: what is this an indicator of? Meaning CycleDiffusion has excellent  SSIM (which  is more correlated to human perception) than baselines, but lower PSNR (actually the lowest). What is this an indicator of? 

 Section 4.3
Am I correct to understand that your contribution in section 4.3 is the fact that you can “work on a latent space” for diffusion models (e.g. apply Langevin) similarly to what one can do with GANs? 
The findings reported are interesting but reporting them without a critical explanation is maybe limiting. E.g “3D GANs are worse at guidance than 2D GANs and diffusion models”: why so? Or at least, what’s your opinion on this?



clarity,_quality,_novelty_and_reproducibility: In my opinion, the clarity of the paper could be improved. I find that many sentences are vague e.g. " a common latent space 'emerges' from...". It could be more streamlined and direct what the point is. 

In terms of novelty - the main contribution seems to be a way of defining a Gaussian latent space + DPM-Encoder for diffusion model and to exploit this perspective for unpaired image translation, zero shot image-to-image editing and conditioning the generation.

In terms of reproducibility, code is not provided in the submission but details of experimental setup are somewhat comprehensive.

summary_of_the_review: I think the paper could improve in its clarity. The novelty seems pretty limited, although CycleDiffusion used as zero-shot image-to-image editor is interesting in my opinion.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper considers a concatenation of all intermediate variables through a forward diffusion as a latent code.

The authors consider the forward process of diffusion models as an encoder.

Using two independently trained diffusion models, the proposed method encodes an image with a fixed random seed for one model and run reverse process on the other model to achieve image-to-image translation.
strength_and_weaknesses: 
Strengths

(+) This paper shows interesting application of diffusion models.

(+) The proposed latent code (concatenation of all variables) is versatile across different models.


Weaknesses

Please refer to the below section.
clarity,_quality,_novelty_and_reproducibility: 
### Clarity

(-) Abbreviation without definition: DPM

> 3.2. DPM-Encoder: an Invertible Encoder ...

> Our contribution is DPM-Encoder, and invertible encoder ... 

(-) Considering an existence of a generator as invertibility of an encoder does not make sense. Generally encoder is considered as inverting a generative process.

(-) Broken sentence: several researchers and practitioners have found that when trained with the same "random seed" [missing comma] [missing subject] leads to similar images. 

(-) Unclear description: Does above sentence mean that the same $X_T$ and noises leads to similar images across different diffusion models?

(-) Missing definition of "zero-shot". In what aspect is the CycleDiffusion zero-shot?

### Quality (Technical soundness)

(-) Why is generally missing.
* Why is the latent code defined as a concatenation of $X_T$ and intermediate noises?
* Why do the models trained with the same random seed lead to similar images?
* Why does PSNR between a translated image and source image make sense?
* Why does FID between generated and target images make sense? What are generated and target images? In the last sentence of the first paragraph of 4.1 says the target image is the generated image.


(-) Guidance in 3.4 is same with usual guidance in diffusion models.

### Novelty

(-) Latent space = a concatenation of $X_T$ and intermediate noises is trivial.

(-) Encoder = forward process is trivial.

(-) Fixing everything and changing text is trivial.

summary_of_the_review: This paper shows interesting phenomena but does not contribute to building general knowledge with logical development.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies the latent space of diffusion models compared to a variety of other generative models such as VAEs, Normalizing Flows, and especially GANs. First of all, this paper provides a very comprehensive and nice overview of latent space across deep generative models and motivates the study of latent space in diffusion models. Then, it traces down to the question about the latent space of stochastic DPMs rather than other methods which leverage "fixed" processes. The authors leverage the finding such that the randomness in the model captures the image-specific information and further utilize the defined latent space for an unpaired image translation task. It explores more tasks such as zero-shot image-to-image editor and latent space guidance with energy-based models.
strength_and_weaknesses: Strength:
* This paper is mostly well-written with a clear literature review, leading to its motivation. 
* The studied problem is important, it is significant to understand the latent space of diffusion models as it has been done in GANs and other generative models. It can lead to more understanding and empirical applications of diffusion models.
* The experimental results empirically support the claims and findings of the paper. 

Weaknesses
* Even though the motivation to study the latent space of diffusion models and how this paper studies it with a simple yet effective approach is well-written, the applications to work as a CycleDiffusion, zero-shot image-to-image editing and latent space guidance sounds a little indirect to me, though this may be due to my biased perspective. I directly thought about GAN inversion and related literature, e.g. [1-3], after I read the motivation of this paper. I don't mean the applications in this paper are not okay, but I would suggest making the motivation for the applications tighter.
* As a long-standing problem in understanding the latent space of deep generative models, even though this paper achieves great experimental results, I would still suggest the authors study theoretical guarantees of it.
* The proposed D-CLIP score is a bit confusing, is the relative change meaningful without absolute value?
* Unified Plug-and-Play Guidance is a bit misleading, why would the proposed or defined latent space unify diffusion models and GANs?

[1] Härkönen, E., Hertzmann, A., Lehtinen, J. and Paris, S., 2020. Ganspace: Discovering interpretable gan controls. Advances in Neural Information Processing Systems, 33, pp.9841-9850.
[2] Shen, Y., Yang, C., Tang, X. and Zhou, B., 2020. Interfacegan: Interpreting the disentangled face representation learned by gans. IEEE transactions on pattern analysis and machine intelligence.
[3] Shen, Y. and Zhou, B., 2021. Closed-form factorization of latent semantics in gans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 1532-1540).
clarity,_quality,_novelty_and_reproducibility: Overall, this is a good paper studying an important problem. It demonstrates technically sound formulation and reasonable results. All my concerns are listed above.

The code is not uploaded, but the approach looks simple, the hyperparameters are given in the appendix, but I hope the authors keep the words to make the code public upon acceptance.
summary_of_the_review: Overall, my main concern about this paper is the flow of the paper and solid experiments to support the method empirically as theoretical results are missing or challenging. I value the novelty of this paper to formulate this important yet not fully explored problem (latent space of diffusion models). I am leaning towards accepting this paper.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes to store the random noises along the generative trajectory of diffusion probabilistic model as a latent code, and use this latent code for image-to-image translation.
strength_and_weaknesses: Strength:
(1) The paper provides a clear summary for previous works.
(2) The paper is easy to digest.


Weaknesses:
(1) Limited Novelty: DPM-Encoder, which stores all the noises along the generative process, is an obvious way to encode an image to the latent space of a stochastic DPM. Since it is obvious, it appears in the famous work [1], Appendix B.2, to interpolate between images. Therefore, the encoding strategy is not new to me.
(2) Limited Novelty: CycleDiffusion is a simple generalization of SDEdit [2] and the observation in [3]. The authors made no additional theoretical justification on the method, making its improvement over previous works limited.
(3) Unsatisfying results: In Figure 3, many of the parts in the images, which we do not expect to change, changed. For example,  the astronaut figure on the bottom-left changes not only in the style, but also the content. This, in my opinion, is not fixable with the current method. 

[1] Improved Techniques for Training Score-Based Generative Models, https://arxiv.org/pdf/2006.09011.pdf
[2] SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations https://arxiv.org/abs/2108.01073
[3] Understanding DDPM Latent Codes Through Optimal Transport https://arxiv.org/abs/2202.07477

clarity,_quality,_novelty_and_reproducibility: See above section.
summary_of_the_review: The paper leverages an existing strategy to encode images generated by DPM, then directly apply this latent code to another DPM to achieve image-to-image translation. The novelty is limited and the results are not satisfying.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper propose a unification to the latent space of diffusion models. This unification is then used to methods like a cycle diffusion. The idea for a latent space for stochastic diffusion probabilistic models is to concatenate all the noises used to diffuse together into a single vector. Keeping these noises level fixed in another generator allows for images to be translated from one distribution to another. That is, cycle generation. Overall, the performance is intriguing, though the presentation and the need for and execution of the unification is unclear and clunky. The presentation really led to the difference in opinions between reviewers. For example, after reading the author reply, reviewer JmnA writes 

" I find the explanation at the end of page 4) a bit obscure and the formalization stretched - it doesn't seem to add much to the intuition in my opinion," 

which is supported by reviewer NUmH in reply to the authors

"I also prefer simple methods than complex ones but they should be supported by some principle."

The results are nice, but the entire presentation could be simplified to only stochastic diffusion probabilistic models and enriched by going into more detail on why the concatenation is the correct choice with the extra page or so of space that would be freed. 
justification_for_why_not_higher_score: The presentation and the need for and execution of the unification is unclear and clunky.
Decision: Reject