Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This work introduced an energy-guided stochastic differential equation based method for Image2Image translation task. Unlike previous methods, the proposed one takes the source training data into account and designs the energy function to preserve the domain-independent features and discard domain-specific features. Experiments on faces and animal faces show the effectiveness of the proposed method over existing GAN based and diffusion based I2I works. 
strengths_and_weaknesses: + A nice interpretation of the discretization of EGSDE in the formulation of product of experts

Please find my concerns below:

(i) My biggest concern is that I’m not sure if energy or diffusion based models have beaten GAN based approaches, as claimed by authors. The most typical GAN based approach is StarGAN [8], which is cited by authors and the dataset source used in this work. However, I do not find any comparison with [8] in the experiment section? Any special reasons? Though [8] handles multi-domain translation, it is still okay to train the framework of [8] with just two domains. Additionally, the FID numbers shown in [8] are much lower than what is shown in this paper. Overall, I’m not quite convinced if GAN based approaches have been outperformed as they also take both source and target training data into consideration.

(ii) Is the proposed method based on energy function much slower than a feedforward step in GAN approach? If so, I probably expect even higher performance in terms of quality. User study is another good way to evaluate perceptual preference if there is no GT to evaluate.

(iii) It looks the EGSDE is not limited to I2I task. Would it be also applicable to other tasks like colorization or inpainting?


--------------
Thanks for the rebuttal from authors. After checking comments from other two reviewers, I increased my score to 5. The reason why I asked authors to compare with the GAN-based approach StarGAN v2 is that authors need to support the claim with solid experiments by saying the diffusion model has unique advantages on this task, not just because diffusion model is popular nowadays and has not much explored for I2I yet. Especially considering that authors have used the dataset from StarGAN work but eventually ignore the comparison in the draft. Be more rigorous and hopefully do not make this type of mistake again in future research.

questions: Check my concerns above.
limitations: Yes.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The author proposes an unpaired image-to-image translation method using score-based diffusion generative models. Compared with existing arts, the method also considers the training data in the source domain. To be specific, the method deploys an energy function pre-trained across two image domains. Experiments show the superiority of the proposed method.
strengths_and_weaknesses: Strength: 
1. This paper considers the domain-independent and domain-dependent features in diffusion models, which is not explored until recently.
2. Measure content similarity with energy function in the diffusion model is a good idea. The authors also provide a theoretical insight into EGSDE.
3. The comparisons are sound and comprehensive.

Weaknesses:
1. One state-of-the-art image translation is missing. Please compare EGSDE with DiffusionCLIP [1].
2. There are many other ways to address image translation and content preserving. One can use classifier guidance to generate images in the target domain [2]. To preserve identity information, one can use identity features. The author should highlight the novelty or the advancement compared with these solutions.
3. The paper will benefit from a more exhaustive ablation study. For example 1) the effectiveness of E_s and E_i. 2) replacing E_s with classifier guidance [2] or classifier-free guidance [3].
4. The paper will benefit from a human evaluation. i.e., comparing EGSDE with SDEdit and ILVR on Male -> Female.

[1] DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation   
[2] Diffusion Models Beat GANS on Image Synthesis   
[3] Classifier-Free Diffusion Guidance


questions: Questions:
1. Please give the detailed implementation of E_i.
2. How about denoising from a perturbed image using DDIM reverse sampling instead of denoising from random noise?

limitations: The limitations and potential negative societal impact have been adequately addressed.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes Score-based diffusion generative models (SDGMs) that does not ignore the training data in the source domain, leading to improved solutions for unpaired I2I. The proposed method is  energy-guided stochastic differential equations (EGSDE) that employs an energy function pretrained on both the source and target domains to guide the inference process of a pretrained SDE for realistic and faithful unpaired I2I.
strengths_and_weaknesses: **Weakness**
- Evaluation on more standard datasets is missing.

**Strengths**
- Intuitive idea, well-written paper.
questions: None
limitations: No immediate negative impact
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper studies SDGMs for image2image translation. The authors propose energy functions pretrained from both source and target dataset to guide the SDE inference process, to generate realistic and faithful images. Experiments show that the proposed method outperforms SDGM-based methods quantitatively.
strengths_and_weaknesses: Strengths: 
1. Introducing the source domain knowledge to improve SDGM-based methods is promising and a good contribution.
2. Quantitative experimental results are rich. The selection of baselines and evaluation metrics is reasonable.
3. The paper is overall well written.

Weaknesses:
1. The background of SDGMs is a bit hard to follow. I suggest the authors give an intuitive understanding of what each equation does for image2image.  Also, I can't find what is f(y,t) in practice.
2. The paper lacks some qualitative analysis about the ablation of each expert.
questions: 1.  What is the drift coefficient f(y,t) in practice?
2. Is sampling a start point $q_{M|0}(y_M|x_0))$ from $x_0$ still necessary, since a faithful expert is introduced? Start point and the faithful expert, who is the main contributor to faithfulness?
3. How is the computation cost of EGSDE, compared with SDGM-based baselines and GAN-based methods? Are the qualitative results elaborately picked or randomly selected?
4. I suggest the authors to present some qualitative results about the ablation of experts and start point.
limitations: A limitation is that, unlike the realistic expert,  the faithful expert simply uses a low-pass filter without knowledge of source and target datasets. As said in the paper, this could be improved in future work.
review_title: null

Meta-review: metareview: The paper proposes an unpaired image-to-image translation method based on score-based diffusion models. Compared to prior works [7, 29], the paper adds two energy functions pretrained on both the source and target domains in an expert-of-product framework. The paper has received positive reviews. Reviewers found the paper well-written, the idea intuitive, and the experimental results comprehensive.  The rebuttal further addressed the concerns regarding the user study, running time, and missing comparisons. The AC agreed with the reviewers’ consensus and recommended accepting the paper. 


confidence: Certain
Decision: Accept