Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Quality：
The proposed general-purpose framework for modeling CMM spaces is worthwhile and insightful. By using a set of domain-agnostic regression cost functions instead of the adversarial loss, it improves both the stability and eliminates the incompatibility between the adversarial and reconstruction losses, allowing more precise outputs while maintaining diversity.

However, it would be interesting to see the qualitative and quantitative comparison with the latest related works. For example, the following two CVPR2020 papers(For reference only):
[1] Zheng C, Cham T J, Cai J. Pluralistic image completion[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019: 1438-1447.
[2] Li J, Wang N, Zhang L, et al. Recurrent Feature Reasoning for Image Inpainting[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020: 7760-7768.

Clarity：
The paper is very well-written and well-structured and is friendly for readers to understand. It starts off by pointing out the key shortcomings of the use of a combination of reconstruction and adversarial losses and then clarifies the framework proposed in this paper for modeling CMM spaces. Continuously, it explains the drawbacks of conditional GAN methods and illustrates the idea via a toy example. After that, with an extensive set of experiments, this paper experiment on several such tasks with different datasets. It illustrates the outperformance in both qualitative and quantitative experimental comparison and demonstrates state-of-the-art performance.

By the way, it may be more friendly for readers to read the paper if the layout of diagrams and tables can be reformatted in a clearer way. 

Originality and Significance：
Due to the ill-posed nature of conditional generation in multimodal domains, this paper proposed a novel generative framework with a simple and generic architecture instead of the adversarial loss. The proposed approach demonstrates faster convergence, scalability, generalizability, diversity, and superior representation learning capability for downstream tasks. At the same time, the comparable performance has been validated on different datasets both quantitatively and qualitatively. And in most of the experiments, it achieves state-of-the-art performance.

Based on the above considerations, I think it is a good paper that marginally above the acceptance threshold. And it may be worthwhile to be accepted if more latest experimental comparison can be shown, and still have the outperformance.

-----------------------------------------------------------------------------------------------
Compared with the latest related work, the author added the qualitative and quantitative comparison against "Recurrent Feature Reasoning for Image Inpainting" in the image completion task. And it shows the outstanding performance of this paper. 
They have also reformatted the paper so the paper becomes clearer for readers to read. 
Consider the above all, I think it’s a good paper and is worthwhile to be accepted.


So I improving my rating to “7: Good paper, accept”

review_title: Conditional Generative Modeling via Learning the Latent Space

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: 
In this paper, the authors proposed a general-purpose framework for conditional generation in multimodal space. The proposed method is optimized to find multi-modal optimal solutions at inference time. This general method can be applied to a lot of down-stream tasks, improving inference performance without the need to carefully design a network structure.

Pros:
1. The proposed method is very general. It is evaluated on quite a few different tasks, including sketch to image generation, image imputation. image colorization, etc.
2. The proposed method shows superior results both qualitatively and quantitatively compared with baseline models. It is quite impressive how comprehensive the evaluation is. The multi-modality perspective is especially well presented.

Cons:
1. The authors should carefully audit the paper. There are quite a few typos and presentation flaws, which makes it much harder to read. For example, the numbering of the figures is not consistent. There is not figure 2, 4, 6, 8 etc. 
2. I understand that the authors want to show as many results as possible, but the presentation becomes very crowded, especially on page 8. I would recommend the authors to leave the critical experimental results here and move some to the supplementary materials.
3. Despite the impressive experimental results, I think readers would benefit more from a deep discussion. For example, what is the relationship between this model with a VAE model? The similarity is of course the continuous latent space. By varying the latent factor, VAE models can generate diverse data as well. 
review_title: Good Model, the Paper Can Be Improved

final_score: rating: 10: Top 5% of accepted papers, seminal paper
review_content: This paper proposes a family of cost functions and a framework for modeling a continuous multimodal (CMM) space.
The proposed model converges more stably and faster than conventional methods and shows high-quality results in several tasks. Also, this method can generate diverse outputs at inference with a single model. It was partially possible with many GAN methods, but there is a significant improvement in diversity.

- Clear motivation and well-defined method\
The problem of modeling the CMM space is well defined, and the limitation of the previous methods are described in detail. And the intuition to resolve this problem is also highly convincing.

- Various and extensive experiments\
The experimental settings and results support the effectiveness of the proposed method. The toy example in Figure 3 clearly shows that the proposed method can model CMM space properly.
Each experiment for downstream tasks also has a detailed explanation, showing good qualitative and quantitative performance.

There are no major weaknesses in the overall content. However, it is necessary to correct that table 5 and table 6 are overlapped, and many figures and tables are mixed in a somewhat disorderly manner.

It is somewhat similar to [1,2] in that a separate variable is defined for generating multiple outputs in a deterministic function. It's good to discuss this.

It would be great if there is a follow-up study to see if it could be extended to generate an image from random variables, like Conditional GAN. And some naming is required to represent the method.

[1] Auxiliary deep generative models, L. Maaløe, ,et al.
[2] Sym-parameterized Dynamic Inference for Mixed-Domain Image Translation, S. Chang, et al.
review_title: A new method for modeling continuous multimodal spaces

final_score: rating: 7: Good paper, accept
review_content: The paper addressed the deterministic inference issue and enabled the conditional generation in multimodal spaces. The authors also explained the `` 'generalizability' advantage over cGANS, that is capable of learning more task-agnostic representations.
This paper is very dense. The authors conducted many experiments to show 1) the presented conditional generative modeling approach is capable of learning diverse representation and hence lead to diverse generation, 2) the proposed methods worked quite well across different tasks, both subjectively and objectively. Ablation studies are also performed to show ```'momentum' as a supplementary aid” is helpful.

At this moment, I don’t have major concerns regarding this submission. I incline to accept this paper, and am willing to further change my rating. Below are some minor comments:
1.	Based on the description in section 2 to 4, and also the strong experimental results, I am convinced that the proposed approach converges faster than cGAN, and can learn diverse representations and enables multimodal generation. However, it is a bit surprising/interesting that proposed method worked consistently better than other prior methods in almost all the downstream tasks explored. Can you elaborate more on why learned representations worked consistently well in almost all downstream tasks that have tried in the paper.
2.	The experimental study part is very dense. It would be good to have a short section clearly compare the model size/capacity of your models and the counterpart models. Also, is it good to move at least one model architecture you used to main text?
3.	The authors can consider comparing and contrasting with the normalizing flow related work. When normalizing flow applied for inference/learning latent representations, it seems (weakly) related to your design described on section 2.1

review_title: An interesting paper for conditional multimodal generation

Meta-review: The paper proposes a model and a training mechanism for multimodal generation. The reviews are generally positive: they praise the generality of the method, the extensive experimental evaluation, and the good empirical results. Overall, no major concerns were raised, and all reviewers recommend acceptance.

A couple of concerns remain, in my view:
- The method is generally heuristic, and intuitively rather than theoretically motivated. This is compensated of course by the empirical evaluation, which is thorough.
- The paper could be better written. The reviewers suggested some minor improvements which were implemented in the updated version, but I believe there is room for further improvement.

Due to the above concerns, I consider the rating of reviewer #3 (10: Top 5% of accepted papers, seminal paper) to be unjustifiably high. On balance, however, I'm happy to recommend acceptance.

Message to the authors:

In the abstract you write: "a simple generic model that can beat highly engineered pipelines". Please be aware that the word "beat" evokes competition, winners and losers, so it's not appropriate in the context of scientific evaluation. Please consider replacing it with something neutral, such as "a simple generic model that can perform better than ...".
Decision: Accept (Poster)