Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper introduced a novel method for unsupervised image-to-image translation with density changing regularization. Different from most recent work, which either assume a cycle consistency or employ contrastive learning, the paper proposed a simple yet efficient method that adds density distribution regularization atop a simple GAN model. More specifically, the density changing regularization is to minimize the change of patch density distribution after translation, where the distribution is estimated from generator layers. The paper provides examples and numerical results which show improvements over baselines on both quality of the images and training speed.
strengths_and_weaknesses: Strengths:
1. The paper is generally well written. The authors provide detailed explanation on methodology and results.
2. The proposed method is simple yet efficient. Such method can be easily replicated to build a strong baseline.
3. The proposed method shows improvement over baselines on most the tasks.

Weaknesses:
1. The results on horse-to-zebra task is not as good as SOTA model. The authors provide potential issue and solution.
2. The ending of the paper is abrupt. While section 6 includes decent discussion on potential issues and solutions, a proper conclusion section is still recommended.

questions: 1. Why choose variance as the distance function between distributions instead of other functions such as KL divergence? It would be good to see the how KL divergence would affect the model
2. The equation on line 121 has no number.
3. The deductions from line 126 to line 133 is not super clean to me, could you provide more details in appendix? I also recommend moving this part to appendix.
4. What is L in line 180?
5. The description of PatchDict in table 3 is too brief. Could you add more details in appendix?

limitations: The authors have addressed the limitations and potential negative societal impact of their work.

I would suggest authors to investigate potential solutions in the future works.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a density changing regularization for better image translation under the assumption that the regions of the high probability density of two domains must be mapped to each other. To do so, the authors define a density constraint as the variance of the density ratio of two domains. Here, they train density estimators for each domain, numerically compute the value, and minimize it to reduce the ratio gap. 
strengths_and_weaknesses: The idea is simple and effective. When the assumption holds, the results show that the density changing regularization improves the translation performance. 


My major concern with this paper is its main assumption. In fact, I could not agree with the argument that the patches with a high (low) density in one domain should be mapped to patches with high (low) density in another domain. I could think of many trivial examples that this is violated (as also noted by the authors). As the authors also noted in the last section, the key to the proposed method is whether the assumptions are met. However, despite its importance, there is no justification for this assumption and no theoretical or empirical observations are supported. The authors simply state the assumption and use it without any explanation, leaving a big logical jump. There must be some experiments or analyses to validate this assumption (at least in part, for the datasets used in the paper). Or, one must verify this with enormous amounts of experiments in various benchmark datasets so readers can agree that this constraint would generally work in many practical situations. If this is not resolved, I cannot give high scores. 

Another issue is the confusing usage of the terminology; “unsupervised”. In fact, this is an “unpaired” setup, not “unsupervised”. Many recent papers have addressed this misusage, the first of which is TUNIT in ICCV 2021. Please revise the term accordingly.


The subscript index “i” is used without defining it. One needs to wait until seeing eq (4) to know what it is. 

There are several typos in the paper: 

Line 104, there are can -> there can

Line 147, {b_i^l}-> {c_i^l},

Line 153 Please note that a^l and c^l represents -> b^l represents.

Line 189 L_mle -> L_nll 

Line 285, Table 2 -> Table 1

It is not clear if the density estimators are trained together or separately. Looking at line 189, it seems like the estimators are trained together. Please clarify this more. If this is the case, wouldn’t this be too unstable? What happens in the early phase of the density estimator working poorly? 

Figure 4 and its description in the “learned densities” paragraph are not easy to understand. How should it be read? It is not clear what the densities under the input semantic maps (first column, second row) in Figure 4 mean (similarly for Truth, last column, second row).

--
After the rebuttal, I changed my score from borderline reject to borderline accept. 
questions: Please see the above comments.
limitations: Please see the above comments.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: In this paper, the authors make a density-changing assumption where image patches of high probability density should be mapped to patches of high probability density in another domain. Then the authors propose an efficient way to enforce this assumption: they train the flows as density estimators and penalize the variance of density changes.
strengths_and_weaknesses: Strengths: 
This paper is well written and easy to understand.

Weaknesses:
1. The results of mAP, pAcc, and cAcc are not consistent with those in the original QS-Attn paper.
2. The results of SWD on both Cat2Dog and Horse2Zebra should be included and compared in Table 1.
3. User study results should be provided.
4. Qualitative ablation results should be provided.
5. More attention-based GAN methods such as [1,2] should be included and compared in Tables 1 and 2.
6. From Figure 3, the proposed method does not significantly improve the performance of the label2city, horse2zebra, and anime datasets.
7. Results of more methods should be provided in Figure 3, especially the label2city and anime datasets.
8. Results of SOTA methods should be provided and compared in Figure 4.

[1]Kim, Junho, Minjae Kim, Hyeonwoo Kang, and Kwanghee Lee. "U-gat-it: Unsupervised generative attentional networks with adaptive layer-instance normalization for image-to-image translation." ICLR 2020.
[2]Tang, Hao, Hong Liu, Dan Xu, Philip HS Torr, and Nicu Sebe. "Attentiongan: Unpaired image-to-image translation using attention-guided generative adversarial networks." IEEE Transactions on Neural Networks and Learning Systems (2021).
questions: See Weaknesses.
limitations: The author does not provide potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a solution for the unsupervised image-to-image translation problem which is based on matching the distribution of generated patches in the output domain Y to that of the input patch distribution in the source domain X. To this end, the paper proposes a loss that aims to minimize the variance between the density of patches in X and the corresponding density in domain Y. The loss is based on density estimation using an auto-regressive flow model applied on patch representation at different layers. The paper demonstrates superiority on a number of datasets in comparison to baseline methods. 
strengths_and_weaknesses: Strengths:
1. The paper is well written and clearly illustrates the motivation behind the proposed solution for the unsupervised image-to-image problem. 
2. The main technical contribution is the density loss. The motivation for using flow models to match the distribution of patches from the input distribution and the generated ones makes sense. I find the simple realization of the main idea through a single loss term to be a strength of the paper, especially as it leads to clear improvements. 
3. The evaluation is relatively thorough, comparing to recent methods and direct alternatives on a range of datasets, both quantitatively and qualitatively. 

Weaknesses:
1. The paper makes an assumption regarding the fact that patch distribution should be matched. While this is discussed in the limitations, I think it should be stated up front and discussed already in the introduction. The method will only work where the assumption holds. 
2. An ablation study into the size of patches would be useful to understand their effect. In the limit, one could consider pixels or the entire image. 


questions: 1. An assessment of the effect of patch size on the output. 
2. An assessment of the limitations, such as examples or analysis of the cases where the assumption does not hold. 
limitations: The authors adequately addressed the limitations and potential negative societal impact. However, it may be useful to further illustrate, qualitatively or visually the cases where the assumption made does not hold.
review_title: null

Meta-review: metareview: This paper addresses the density-mismatch problem in image-to-image translation by introducing a patch-wise variance constraint regularization. The approach is simple and effective, according to the reviewers. There were some general concerns about the validity of the assumption, but the authors appear to have sufficiently addressed those concerns. I would encourage the authors to make it clear that this is an inductive bias that they're relying on to make their method work: it's a valuable contribution but I think it's worth being extra clear that this is a reasonable assumption they built into their model but it might not be the best one. 

I therefore recommend acceptance of this paper to NeurIPS.

There was one negative review from zLZj that had some useful content, but the authors seemed to address those concerns fairly well. The reviewer was showed skepticism towards the method that wasn't entirely clear to me and wanted to look at the code themselves but never followed through. I wasn't terribly convinced by the score being so low after the discussion and the author rebuttals, and I don't see evidence that reviewer looked at other reviews or discussion, so I believe the score does not accurately represent the paper's quality and I will treat the score (the discussion was good) as an outlier.

Both wXXZ and o4oL did well as far as discussion and engagement.
confidence: Less certain
Decision: Accept