Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes asymmetric reverse process (Asyrp) to exploit the semantic latent space of pre-trained diffusion models. With the guidance of CLIP, the proposed method can find semantically meaningful directions in the latent space for image editing. 
strength_and_weaknesses: Strength:
(1) As far as I know, this paper is the first work to directly find the semantic directions in pre-trained diffusion models.
(2) The empirical results look impressive.

Weaknesses:
(1) The choice of h-space needs more clarification. The Figure 15 and Appendix C.3. could not explain everything. My questions are: [1] Are images in Figure 15 training images or test images? [2] If they are training images, why does the layers other than layer 8 have bad performance? I imagine they are all able to minimize the loss in Eq. 7. [3] If they are test images, why does layer 1-6 does not change the generated images at all? At least there should be some changes.
(2) The implicit neural direction looks unnecessary. Could the authors provide more ablation study to compare using $f_t$ and $h_t$? Moreover, I cannot understand the results in Figure 6 (a). Since $\epsilon$ has a higher dimension than $h$, I can hardly imagine it yields higher loss when optimizing Eq. (7). 
(3) There are too many magic numbers in Section 4, making me doubt the generalizability of the proposed generative process. Though the authors claim that these magic numbers work across the five datasets, I still think quantitative results should be provided on analyzing the sensitivity of these magic numbers.

clarity,_quality,_novelty_and_reproducibility: Clarity: Could the authors discuss the relationship between CLIP-guided Diffusion and your theorem 1? It is a little bit weird to me since CLIP-guided diffusion does work, but your theory suggests the contrary? 

Quality & Novelty: See the above section.

Reproducibility: The authors provide enough details in the appendix for reproduction. I think I can reproduce it by myself.
summary_of_the_review: Overall, the goal of the paper is intriguing, but the paper contains a number of heuristics without enough strict ablation study and quantitative results, making the claims doubtful.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper discovers a semantic latent space (termed h-space) for pretrained diffusion models. The proposed h-space is a shift/residual in the middle-layer feature of the UNet, and is discovered via finetuning. The authors propose asymmetric reverse process (Asyrp) for both finetune and sampling. The proposed h-space has several nice properties: homogeneity, linearity, robustness, and consistency across timesteps.
strength_and_weaknesses: ## strengths
- Finding a semantic latent space for pre-trained diffusion models is an important problem and is relatively less studied. The proposed Asyrp is a promising endeavor in this direction.
- The proposed h-space has several nice properties: homogeneity, linearity, robustness, and consistency across timesteps.
- The empirical study in generative process design suggests an interesting analysis for editing flexibility.

## weakness and detailed questions
- Theorem 1 seems a little bit weak to me, it says that $\Delta x_{t-1} \approx 0$ given a shift $\Delta \epsilon_t$. However, this conclusion is trivial given the well-known result from DDIM Eq. 13. The resulted loss function Eq. 7 seems same as the GPU-efficient finetuning in DiffusionCLIP [1].
- In Eq. 7, is $x_t^{edit}$ $P_t^{edit}$?
- Can we think of the proposed training of Asyrp as a constrained version of DiffusionCLIP (only the middle layer of UNet can be finetuned)? If so, how does the generative process of Asyrp compare to DiffusionCLIP, i.e. unmodified (symmetric) DDIM sampling?
- Linear combination (Figure 8) is quite interesting. How does this compare to a linear combination of scores (as in DiffusionCLIP)? This comparison would be interesting since it saves multiple UNet forward passes.
- This won't impact my rating. I am curious is it possible to extend the proposed h-space in text-to-image diffusion models such as StableDiffusion?

[1] Kim, Gwanghyun, Taesung Kwon, and Jong Chul Ye. 2021. “DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation.” arXiv [cs.CV]. arXiv. http://arxiv.org/abs/2110.02711.

clarity,_quality,_novelty_and_reproducibility: - The paper is well-written for the most part.
- The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
- Results should be reproducible as the code is provided.
summary_of_the_review: I am willing to amend my score is my major concerns are addressed.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors introduce an algorithm that manipulates images using diffusion models.
They do so by manipulating the representation in the bottle-neck U-net layer of the diffusion model over several (but not all) timesteps. The loss function that they optimize is the same as for DiffusionCLIP.
strength_and_weaknesses: ### Strengths

The visual results are striking. There are plenty of examples of different attributes for editing. A user study (n=80) was conducted and showed very clearly superior performance to DiffusionCLIP. The algorithm can be applied to pretrained diffusion models.


### Weaknesses
Theorem 1 seems to be formulated a bit sloppy... $ \tilde{x}_t \approx  x_t $
isn't really saying anything. Defining a bound $||\tilde{x}_t -  x_t|| \leq \delta$ and then showing that $\delta$ is small would be better suited.
For the proof in appendix B, I would prefer if all symbols were defined in that section. In equation (13) you introduce $\beta$ and $\eta$ without stating their relation to previous terms.
In general, theoretical motivation for the procedure and parameter choices is lacking a bit.

The only quantitative evaluation is the user study. Results for DiffusionCLIP seem a bit lower quality than in the original paper.


clarity,_quality,_novelty_and_reproducibility: ### Novelty
Traversing the latent space of a diffusion model causing semantic changes by manipulating the image representation is a novel contribution.

### Clarity
The procedure, though a bit convoluted, is clearly explained, and graphics support the understanding of the algorithm. Images in Figures 1 and 2 could be enlarged a bit to ease readability.

### Reproducibility
The algorithm is in pseudo-code in the appendix and the authors also provide code that will be published with their contribution. I have not checked the code, but if functional, reproducibility is provided.

### Quality
The results look great. The procedure is nicely explained. I also appreciate the background section 2.

### minor mistakes
There are some minor language mistakes. For example:
- in 2.1: that learns data distribution $\rightarrow$ that learns a data distribution
- in 2.1: is variance schedule $\rightarrow$ is the variance schedule 
- in 3.1: given a pretrained and frozen diffusion models $\rightarrow$ given a pretrained and frozen diffusion model
- in 6: image editing on semantic latent space $\rightarrow$ image editing in the semantic latent space
- in A:  On the other hands $\rightarrow$ On the other hand
- in A:  incorporating diffusion models with scorebased model $\rightarrow$ incorporating diffusion models with scorebased models
- in A:  However it requires noise-dependent $\rightarrow$ However it requires a noise-dependent
- ...
summary_of_the_review: Overall, I think this is a nice contribution that carefully explains the authors' procedure and provides visually striking experimental results. The contribution is novel and presents a significant step towards practical image editing with diffusion models. With additional theoretical justifications, this would be even better. I would recommend it for publication at ICLR.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposed a new method called Asyrp (asymmetric reverse process) to discover the semantic directions in the latent space of pre-trained diffusion models. Specifically, this work first theoretically and empirically showed that a shift added to the UNet noise prediction results in almost the same reverse process trajectory. To this end, an asymmetric reverse process is proposed to manipulate the real image by controlling the latent variable. Besides, this work found that the bottleneck (or the deepest) feature map in the UNet, termed h-space, has the most salient properties than other latent spaces, and thus considered to learn implicit neural directions in the h-space. Finally, in order to achieve the best tradeoff between editing strength and generation quality, this work designed a full editing process, consisting of three phases: editing with Asyrp, traditional denoising, and quality boosting.

strength_and_weaknesses: Strengths:
- This work is clearly written and it is overall very easy to follow. 
- I liked the idea of discovering the semantic directions in the h-space of existing diffusion models, since the state-of-the-art diffusion models use the UNet backbone. Compared with the noise space which has been considered in previous works, the h-space is more compact and contains higher semantic information. Moreover, the work semantically demonstrated in various experiments several nice properties of h-space: homogeneity, linearity, robustness and rough consistency across timesteps. 
- Other important components in the proposed method, including asymmetric reverse process, implicit neural directions, three-stage generative process design, have also shown via experiments the respective effectiveness in contributing to the success of the proposed method.
- The experiments were conducted across different diffusion model architectures and different datasets to show the general applicability of the proposed method, and also demonstrated that it outperformed the previous fine-tuning method DiffusionCLIP.

Weaknesses:
- In the proof of Theorem 1, it assumed that $\beta_t \approx 0$. This might be true only when $t \to 0$. As $\beta_t$ monotonically increases from $t=0$ to $T$, I wonder if this assumption still holds when $t$ is large. I think more justifications are needed here to clarify the possible confusions.
- In experiments, many “less-diverse” datasets have been considered, such as CelebA-HQ, LSUN bedroom/church, AFHQ-dog. I wonder why not consider more diverse datasets, such as ImageNet? In particular, we know diffusion models have achieved good generation results on ImageNet. Compared with previous latent space traversal works based on StyleGAN that cannot work on ImageNet, this result will show a big advantage of the proposed method.
-  Since we already have many StyleGAN-based methods (StyleCLIP, StyleGAN-NADA, etc.) for latent semantic discovery, how does the proposed method based on diffusion models compare with them? I can tell that for the diffusion-model-based method, we still need many more tricks (e.g., asymmetric process, three-stage scheduling, implicit direction, etc.) to make it work reasonably well. I’m not saying that we have to beat those methods in this paper, but even not, it will still be interesting to see the pros and cons of both two methods and to provide guidelines for further improvements.

clarity,_quality,_novelty_and_reproducibility: I’m satisfied with the work's overall quality, clarity, and originality. There are some clarity-related questions: 
- In Sec 3.2, I’m sure if it is reasonable to call $D_t$ the drift, which confuses me by thinking about the drift coefficient in the SDE formulation.
- In Sec 3.4, when we use a custom sub-sequence for the reverse process, why do we choose this normalization $S / \tilde{S}$? 

summary_of_the_review: Overall, I liked the idea of discovering the latent semantics in diffusion models and I think the experiments well support the claimed contributions in the work. Since I still have some concerns about the theory and experiments (see the weaknesses above), I put my initial rating as “weak accept”. I’m open to increasing my score if the rebuttal well addresses my concerns.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The authors propose a method for finding a semantically meaningful latent space in pre-trained diffusion models. The idea is novel, the results are quite impressive, and work for many different architectures and datasets. One minor concern is that the theory (particularly Theorem 1) is a bit weak, but the utility of the method and the impressive empirical results outweigh weaknesses.
justification_for_why_not_higher_score: As noted by the majority of reviewers, the theory is weak relative to the empirical results.
justification_for_why_not_lower_score: All reviewers agreed that the empirical results are both extensive and quite strong. In addition to showing that the method can be used for different architectures across many datasets, the authors also conducted a user study to demonstrate performance superior to DiffusionCLIP.
Decision: Accept: notable-top-25%