Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This work proposes a new method Conditional Hierarchical IMLE to get around limitation of requirements of a large number of samples to generate high-fidelity images.
The proposed CHIMLE is shown to improve generated image fidelity, with a clear reduction in Fréchet Inception Distance compared to the prior best IMLE-based method.
strengths_and_weaknesses: 
Strengths:
To generate samples in a way such that the best sample is about as similar to the observed image as if a large number of samples had been generated without actually generating that many samples,
the author proposes several methods, including partitioning the latent code, partial evaluation of latent code components, and iterative construction of latent code. These ideas give rise to a the method of Conditional Hierarchical IMLE or CHIMLE. 
This work demonstrates that CHIMLE significantly outperforms the prior best IMLE-based method in terms of both fidelity and diversity on a variety of tasks. Besides, they also show that CHIMLE achieves superior image fidelity and mode coverage compared to leading general-purpose multimodal and task-specific methods.



Weaknesses:
1. In the experiments, what backbones are adopted in CHIMLE for different tasks, e.g., colorization, super resolution. If the comparison fair as different methods use different generator structure, e.g., BicycleGAN, MSGAN.
2. Some proper metrics should be used for different tasks, e.g., PSNR for super-resolution.
3. Please check the citation forms, I find some citations are in false format, e.g.:
Yingchen Yu Rongliang Wu Shijian Lu Fangneng Zhan, Jiahui Zhang. Modulated contrast for versatile image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022. 
questions: See above comments.
limitations: The limitations and  social impact have been discussed in the manuscript.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper focuses on "one-shot" (so the SOTA autoregressive transformer and diffusion models are not considered for the comparison in this paper) multi-modal conditional generative tasks. The proposed method is based on the IMLE algorithm. The authors reduce the computational cost (in the training stage) of the previous conditional IMLE framework by a divide and conquer strategy. Experiments are conducted on night-to-day, monocular image super-resolution, image colorization, image decompression, and layout-to-image tasks to validate the effectiveness of the proposed method over the previous conditional IMLE scheme.
strengths_and_weaknesses: *Strengths*
1. The paper is well-written, and the idea of divide and conquer for improving cIMLE is sound.
2. The authors provide a clear introduction to the previous cIMLE method, which is very helpful for readers unfamiliar with this field.
3. According to extensive experimental results, the proposed method synthesizes more realistic and diverse images compared to several baseline approaches.
*Weakness*
1. Several metrics are proposed to evaluate the realism and diversity of the generated images, e.g., precision and recall. Can the authors provide more intuitions why these metrics are not used in measuring the diversity in conditional generative modeling?
- Kynkäänniemi et al., "Improved Precision and Recall Metric for Assessing Generative Models."
2. Recent data augmentations can significantly improve the performance of GAN-based approaches, e.g., ADA, especially in the limited training data case. Can the authors comment on 1) how the proposed IMLE-based performs against the GAN-based methods trained with ADA and 2) how the proposed algorithm performs in the limited training data case?
- Karras et al., "Training Generative Adversarial Networks with Limited Data."
questions: Please see Weakness.
limitations: The authors address the limitations and potential social impact.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper aims to generate diverse and high-fidelity images in conditional image synthesis with a relatively small number of samples. Compared with the previous method conditional IMLE that counters the mode collapse, this paper improves the tradeoff between sampling efficiency and quality by introducing a diver-and-conquer selection mechanism by dimensions in the latent space. The proposed strategy brings better fidelity as well as the diversity of the generated results.
strengths_and_weaknesses: Strength:
+ The proposed divide-and-conquer strategy for latent code selection is very interesting. The division of latent code by dimensions is simple yet effective, and the partial evaluation of latent code components can well address the sub-problems with a modified criterion. When combing the latent code components, this method also considers the correlation between them and proposes the solve the sub-problems from low to high resolutions.
+ The intuition behind this proposed method is clearly explained. 
+ The experimental results show great improvement on multiple tasks (16x super-resolution, night-to-day image translation, image colorization, extreme compression artifacts reduction)

Weakness:
- The divide-and-conquer algorithm aims to solve conceptually difficult problems with improved efficiency and parallelism. However, since this method requires constructing latent code at lower resolutions first, parallelism cannot be achieved. Besides, although the adoption of this time-tested algorithm into latent space sampling is very interesting, the author does not discuss too much about why high efficiency/low cost is vital for this problem. While the author does mention the sampling efficiency of CHIMLE is better than cIMLE, it is not very clear how it connects to the proposed strategy.
- This paper changes both network structure and the sampling strategy, while the experimental results do not include such ablation studies that disentangle the two. In the IMLE paper, the authors at least compare the SRIM and BicycleGAN with the same generator architecture to show the effectiveness of the proposed method, which could serve as a good example for this paper.
questions: 1. As expressed in the weakness part, I have the following concern: does the hierarchical conditioned generation process deprecate the parallelism of this method? 
2. And, how does the sampling efficiency connect to the proposed strategy that uses divide-and-conquer?
3. It is unclear whether the improved network architecture or the CHIMLE method contributes more to the SOTA performance. Can this be verified in new ablation studies that adopt the same generator architecture while using cIMLE, or adopt cIMLE's RRDB structure while using CHIMLE?
limitations: The author has addressed the limitations and potential negative societal impact at the end of this paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents a novel way of applying a non-adversarial image generation technique called IMLE to image-to-image problems.
Namely, the authors propose to split a latent code into several parts corresponding to different image scales. These sub-codes are selected coarse-to-fine: First, the code corresponding to the lowest resolution is found among the randomly sampled candidates. Afterward, this latent sub-code gets frozen, and subsequent parts of the code are chosen. As demonstrated in the paper, this hierarchical approach achieves the same quality as the baseline, with a lower number of generated samples.

According to the reported results of the evaluation, the presented approach outperforms several general-purpose image-to-image models, as well as some task-specific methods.
strengths_and_weaknesses: 1. Strengths.

    This paper proposes an interesting idea of incorporating multi-scale techniques into the IMLE-based generative model.
    This sound quite reasonable and it's a pleasure to see that this trick helps to reduce the number of required candidate samples, which is a common downside of IMLE.
     Moreover, the obtained results show that in the considered setting the presented solution outperforms even the task-specific baselines, which proves that CHIMLE may be useful for applications.
     Also, the authors have submitted the code, which is a good service for the community. 

1. Weaknesses.

    * The explanation is sometimes unnecessarily wordy: while the method finally just employs the multi-scale search of latent codes, it is presented in a much more general way, although only one way of latent division is considered. 
    * The evaluation uses F-measures computed from precision and recall. As shown in [1], these metrics struggle with issues, and it is more common to compute so-called Improved Precision and Recall [1].
    * Unfortunately, the resolution of generated images is not specified explicitly in the paper. Therefore, I am not sure if this approach may be scaled up to the image size of 1K. I ask the authors to provide the image size the model operates. 
    * Why were not models with diverse outputs considered as task-specific baselines? E.g. for the case of super-resolution, SRFlow [2] may be a suitable one. This paper presents a novel way of applying a non-adversarial image generation technique called IMLE to image-to-image problems.


1. References

    [1] Kynkäänniemi et al. Improved Precision and Recall Metric for Assessing Generative Models. NeurIPS 2019.

    [2] Lugmayr et al. SRFlow: Learning the Super-Resolution Space with Normalizing Flow. ECCV 2020.

1. Post-rebuttal comments.

    I thank the authors for their feedback. It has addressed my main concerns. Therefore, I am inclined to increase the score.
questions: I ask the authors to address the concerns listed above.
limitations: The authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

Meta-review: metareview: This paper introduces a conditional image synthesis method based on Implicit Maximum Likelihood Estimation (IMLE). Compared to previous work CIMLE, the paper has introduced a divide-and-conquer method to accurately estimate latent code without evaluating many samples. The paper has received consistently positive reviews. Reviewers found the idea intuitive and interesting, and the method effective (especially compared to CIMLE). The rebuttal further addressed the concerns and included comparisons with the same backbone architecture and additional baselines and evaluations. The AC agreed with the reviewers’ consensus and recommended accepting the paper. 

confidence: Certain
Decision: Accept