Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies the problem of (text/image) guided (single) image editing using a pre-trained diffusion model. Specifically, it proposed a novel diffusion-based image translation method by disentangling style and content representations. Borrowed from the disentangling technique introduced recently (Tumanyan et al., 2022), a pre-trained Vision Transformer (ViT) has been used to aid the style and content disentanglement during the diffusion process. To accelerate the reverse diffusion process, this paper also applied two techniques, namely, semantic divergence loss and resampling strategy (Chung et al., 2022b). Experimental evaluations have been conducted on Animals and Landscapes datasets. Both quantitative and qualitative results demonstrate the strength of the proposed method over existing baselines such as Splicing ViT (Tumanyan  et al., 2022), VQGAN-CLIP (Crowson et al., 2022), and Flexit (Couairon et al., 2022).

strength_and_weaknesses: Strengths:
* [S1] This paper presents a unified framework to improve guided image translation with a pre-trained diffusion model. Results on both text-guided and image-guided applications look more realistic (in terms of shape and texture) compared to the existing baselines, including very recent ones published this year.
* [S2] The combination of different techniques including slicing Vision Transformer, Manifold Constrained Gradient (MCG), and Come-closer-diffuse-faster (CCDF) make a lot of sense. It is great to see a combination of recent techniques really make a big difference on the guided image translation quality.

Weaknesses:
* [W1] Although results are very impressive on the animals and landscapes dataset, the technical novelty of the paper is very constrained. In the Figure 2, almost all the components have been explored in the previous work. To be fair, this paper is the first one that shows great results by combining all of them in a unified framework.

* [W2] As this paper leverages many pre-trained models such as a pre-trained image diffusion model, a ViT model, and a CLIP model, the reviewer feels the comparison to existing baselines in the guided image translation setting is a bit unfair. For example, what happens if we leverage pre-trained models and apply loss functions to aid the SANet (Park and Lee, 2019). It would be good to show some fair qualitative comparisons in the image translation setting (in addition to the Section 4.4). 

* [W3] It is questionable if the guided translation results look very diverse or not. The reviewer would like to know if this is a sign of memorizing the training set (from the pre-trained models). For example, it would be good to find the closest example in the ImageNet and show them on the side. Alternatively, it would be good to show multiple translation results from either image-guided and text-guided applications.

* [W4] The reviewer noticed that FlexIT actually achieved better LPIPS score in Table 1. It is true that FlexIT is trained directly with LPIPS score, as explained in the paper. The reviewer still feels that the proposed model is structure-aware (due to disentanglement) but not very content-preserving. It would be good to elaborate a bit on this.

* [W5] What are the failure cases of the proposed method? It is important to mention the failure cases in the main text and show more results in the supplementary material.
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written with a good amount of details.
While the results look impressive, the novelty is a bit limited as mentioned in the review.
The paper contains sufficient details for a domain expert to reproduce. However, it would be great if the code can be open-sourced in the future.

summary_of_the_review: Overall, I think this is a good paper. I am happy to raise my score if the concerns can be addressed in the rebuttal.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper works on text-guided and image-guided image translations. The authors propose a guided diffusion method that tries to 1) maintain the structure of the source image, and 2) generate semantic content that matches the guidance. Specifically, contrastive loss, CLIP loss, semantic style loss and divergence loss is used during the reverse diffusion process. Experimental results demonstrate the effectiveness of the proposed method over several guided diffusion approaches and some image-to-image translation models.
strength_and_weaknesses: *Strength*
1. The paper is well-written. The authors clearly introduce and detail the proposed approach.
2. The proposed method is shown to be more effective compared to existing guided diffusion methods on the text-guided image translation task.

*Weakness*
1. In common image-to-image translation, "style" referes to the variation which is not shared across different visual domains, which is actually defined in a data driven manner. For example, in AFHQ, style indicates the species, while in winter <> summer, style represents the color variation. Since the two losses (CLIP and semantic style) the authors use only encodes high-level semantic information, how does the proposed method handle the case that style encodes the low-level (e.g., color) variation? For example, the orange sky color is not correctly transfer to the output image in Figure 7 (b). Or more general speaking, how do the proposed method learns to differentiate what is common/unique variation that shared/not shared across different visual domains?
2. Although better visual quality is shown in the image guided translation compared to other methods, diffusion-based model is known for (relatively) long inference time. How long does it take for the proposed method to synthesize an image from an image guidance? How does that compare with other single-shot image translation method?
clarity,_quality,_novelty_and_reproducibility: *Clarity*: This paper is well-written.

*Quality and novelty*: The authors leverage guided diffusion for single shot image translation, and the visual results are more realistic compared to other methods.

*Reproducibility*: I believe the authors provide sufficient detail to reproduce the results in the paper.
summary_of_the_review: Overall, the paper is well-written and the results are somewhat promising. I hope the authors can address the concerns raised in the Weakness section.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a modified Denoising Diffusion Probabilistic Model with a proposed content preservation loss. It aims to achieve image-to-image translation with the content structure preserved and style changed. To do so, a content reservation loss and a resampling mechanism are proposed. Additionally, experiments of text and image-guided semantic image translation are established to verify the effectiveness of the proposed method.
strength_and_weaknesses: Strength:

* The paper is well-written.
* The experiments are sufficient. 
* The motivation for preserving the structure and changing the style is reasonable.

Weakness:
* The goal of keeping content structure unchanged can be verified by segmentation experiments rather than user studies.
For example, you can train a segmentation model on the target domain and then test it on the images generated from the source domain. Since the content structures are preserved, the predicted mask would be close to the mask of source images.

clarity,_quality,_novelty_and_reproducibility: The idea is simple, and the methodology is described clearly, which makes it easy to follow. The experimental studies are sufficient. The novelty of this paper should be above the borderline.

summary_of_the_review: This paper introduces a modified Denoising Diffusion Probabilistic Model with a proposed content preservation loss. It aims to achieve image-to-image translation with the content structure preserved and style changed, which is well-motivated. To do so, a content reservation loss and a resampling mechanism are proposed. The experimental studies seem to be sufficient. Thus, it might reach the publication demands.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a diffusion-based image translation approach. It proposes a new way to disentangle the style and content representations. The content information is represented by the key similarity of a pretrained ViT model, and the  style information is represented by the last layer of CLS token.
strength_and_weaknesses: Strength:
1 The empirical results are pleasing. It outperforms all the existing methods.

2 The authors have conducted extensive experiments on Sec4.4 and the Appendix to support their ideas. 

Weaknesses:

1. It seems that the contribution of this paper is to propose  L_cont and L_sem to represent the content and style information. However, it is not clear why such representation is superior compared with alternatives (e.g., VGG-based ones[1]). It will be interesting if the authors could have a thorough comparison. 

2. there are some typos in the paper, e.g.,  in Fig. 2. The L_con should be L_cont. Please carefully proofread the paper.

[1] Perceptual Losses for Real-Time Style Transfer and Super-Resolution.
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and easy to follow. The authors has conducted extensive experiments. 
summary_of_the_review: Overall, the presented idea looks new, and the performance is promising. The authors have also conducted significant experiments to support their claims. Therefore, I vote for acceptance at this stage.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper was reviewed by four knowledgeable referees. The reviewers found the contributions sound (VSVg) and simple (9VMA), the paper clearly written (VSVg, swjc, 9VMA), the qualitative results impressive/pleasing (VSVg, CBU), and the experiments rather convincing (9VMA, swjc, CBU4). The reviewers raised concerns w.r.t. the novelty (VSVg, 9VMA), the comparisons with methods which do not use pre-trained models (VSVg), the missing comparisons with losses based on different representations (CBU4), and the limitations (which were not discussed) (VSVg). The authors engaged in discussion with the reviewers and adequately addressed all their concerns. After discussion, some reviewers still find the novelty limited (combination of existing techniques) but they do agree that this paper is the first to show great results with those techniques, and has the potential to stimulate further research in this direction. Therefore, the reviewers lean towards acceptance. The AC agrees with the reviewers and recommends to accept. The AC strongly encourages the authors to include the reviewers feedback in the final version of their manuscript - in particular, toning down statements w.r.t. achieved diversity, including the limitations (failure cases) discussion in the main body of the paper, and adding the reported results without the pixel-wise l2 loss. 
justification_for_why_not_higher_score: The paper tackles a relevant problem, appears well executed, outperforms the baselines, and shows great qualitative results, which could motivate additional research. The novelty of the paper is however perceived as limited as the paper leverages previously explored techniques.
justification_for_why_not_lower_score: The paper tackles a relevant problem, appears well executed, outperforms the baselines, and shows great qualitative results, which could motivate additional research.
Decision: Accept: poster