Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Problem: The paper deals with the problem of pluralistic image completion. This means that we want to generate multiple possible complete images.

Methodology: The authors propose a method which uses probabilistic graph models for the problem. This allows to handle visual reality and diversity explicitly by adding constraints and using a Gaussian mixture model (GMM).
 
Experiments: The proposed method produces good(most often state-of-the-art) results on various datasets like CelebA-hQ, Places2 and ImageNet both in qualitative and quantitative terms.

########## POST REBUTTAL ###
I have updated my rating as the authors have addressed my 2 key concerns.
strengths_and_weaknesses: Strengths:
1. Problem Relevance: The problem is quite relevant because diverse generation is more important than unique generation.
2. Quality: I like the way the method section has been structured. Starting from the Objective ( sec 3.2) is a great idea. Because this allows future work to also build up on the objective and come up with different ways to solve it.
3. Clarity: The paper is well written and clear.

Weaknesses:
1. Missing ablation studies

questions: 1. Please don't use the phrase 'Mixture of experts' because mixture of experts means something else in deep learning these days. It refers to models like LiMoE and Switch Transformer[B]. I don't see why you cannot just use GMM because that is all you are using. Can the authors please give a justification as to why this term is used?
2. Ablation studies: Can the authors provide some ablation studies that will help use understand the contribution of the different components? For instance, the effect of the different losses used.


[A] LIMoE: Learning Multiple Modalities with One Sparse Mixture-of-Experts Model

[B] Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
William Fedus, Barret Zoph, Noam Shazeer
https://arxiv.org/abs/2101.03961 


limitations: No, not at all. I am quite surprised that a GAN paper is not discussing negative impacts despite all the deep fakes that have been propagation around and bringing bad publicity to AI.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes an end-to-end probabilistic method. The authors introduce a unified probabilistic graph model that represents the complex interactions in image completion. The entire procedure of image completion is then mathematically divided into several sub-procedures, which helps efficient enforcement of constraints. The sub-procedure directly related to pluralistic results is identified, where the interaction is established by a Gaussian mixture model (GMM). The inherent parameters of GMM are task-related, which are optimized adaptively during training, while the number of its primitives can control the diversity of results conveniently. The proposed method reaches state-of-the-art performance on various datasets.
strengths_and_weaknesses: Strengths:
- The theoretical proof is well conducted.
- Experiments are convincing


Weakness:
- The shown figures are mostly low-resolution. Can the proposed method generate high-resolution images?
- Lack of comparison with other generative models. E.g., GAN/VAE-based or diffusion models.
- What is the advantages of using Mixture-of-Experts to do variational inference compared to traditional variational models (e.g., VAE)?

questions: See above.
limitations: See above.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: For pluralistic image completion, the paper presents a probabilistic method that introduces a Gaussian mixture model(GMM) to build the interaction of some variables in image completion procedure. The GMM whose parameters are task-related is beneficial to the diversity and reliability of completion results. The method is evaluated on five datasets, i.e., CelebA-HQ, FFHQ, Paris StreetView, Places2, and ImageNet.
strengths_and_weaknesses: Strengths:

+The probabilistic method is interesting, its theoretical derivation and implementation steps are easy to follow.

+The proposed method is effective on the experimental datasets.

Weakness:

-The motivation for this paper is a bit hard to follow (in Line 5-7 and 32-41). Specifically, what is the constraints for visual reality? What does the word "task-agnostic" mean,  in this particular scenario, what is the specific meaning of "task" in this term? It's better to explain the in more details.

-The paper lacks more discussions about the correlation and differences between its own method PICME and existing works for pluralistic image completion in Line 110-114. Meanwhile, the experimental tables lacks the comparison results of some cited methods, such as [39,25], etc.

-About the Mixture-of-Experts Model, the size of the number of primitives k is not specifically discussed and analyzed, so the difference in effectiveness between the single Gaussian model when k=1 and the GMM when k>1 cannot be verified.

-The paper has some typos and should be checked carefully, such as the lack of a predicate in Line112 and lacking quoting symbols in the experimental tables.

questions: Although I think this paper is an overall interesting paper, and most the experimental results are effective, I still have some concerns about this paper (more details could be found in weakness):

-the motivation  is a bit hard to follow;

-some more comparison to the related works is required;

-some details of evaluation are also required;

-the writting also could be improved.  
limitations: The authors claimed that the limitation is described in the checklist, however, I didn't find the explicit discussion about the limitation. I find some comparison to ICT, which obtains better results than the proposed method, due to the using the larger model with transformer architecture.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents a nice theoretical framework to provide a diverse set of plausible images that come from the reconstruction process under occlusion. The proposed approach aims to deal with diversity by using  a mixture of experts that are based on a mixture of Gaussians. Several comprehensive experiments seem to testify the usefulness of the proposed approach.
strengths_and_weaknesses: Strengths
This paper contributes to the pluralistic image completion by using a grounded theoretical framework. This is achieved in an elegant way by using a GMM approach as a way to meet diversity. Also, this framework seems to be a generalization of [52], where the latter is focused on only one mixture component of the Normal distribution (unimodal Gaussian) with predefined set parameters  
Weaknesses
Although I think this is a valuable contribution to this research field, I have some questions that perhaps constitute the main weaknesses of the present paper.

questions: - In section 3.2 it is not so clear how precisely the 1-step Monte Carlo (MC) is done. MC is a way to obtain samples from a distribution in an iterative way. Such an iterative process is not clear. The term “1-step”, refers to what, precisely …
- Also, and related to the above and the MC question, how could it be possible to introduce the Expectation Maximization (EM) algorithm to determine the mixture components of the GMM. Or a sort of …It is well known the EM is able to estimate such components, where the means and the variances of the GMM components are iteratively updated. As it is written in the paper I can figure out the “iterative” update process. Is this done in a single one-step? Is it for this reason that the authors call the algorithm “1-step” ?    
- Perhaps, this is the reason why I did not follow exactly the difference \hat{z} and z_c,  … and how these samples are iteratively obtained.
Concerning the above remark, I would recommend to include a figure regarding the architecture of the approach, to better see how the variables play in the framework. I miss the big picture here …
- Determining the number of the mixture components (i.e. “k”) is a model selection problem. Again, taking the EM as a reference procedure, it is known that it is possible to determine the order of the model by using, for instance, the minimum description length. This issue is not addressed in the paper. So the question is how to handle this problem, that is the estimate automatically “k” in an end-to-end methodology as proposed herein.
- Seems that all the components of (2) are modeled by using losses (I’m not sure if this is something related to the above question).  Thus:
-term a) is modeled using L_R + \lambda_A * L_A
-term b) is modeled using L_GMM = LF + L_BM
-term c) is models by using L_ELBO
From the above I did not follow (and did not understand well) the impact of the first term in L_A in eq. (7)

- Another issue that is not clear is written in lines 210-215. By the way, a difficult paragraph to follow.
In the mentioned paragraph it is said that:
- \hat{I^j}_m is generate from  \hat{z^j}_c (line 210)
- \hat{I^j}_m is generated from \hat{I^j}_o (line 211)
So, \hat{I^j}_m has two different sources of generation ? I miss something here 
-It would be nice to introduce a “break point” from which the algorithm stars to fail, and to relate this to the GMM …

Minor comments:
I think the acronym “PGM” (line 69) is not introduced previously.

limitations: See my questions above.
review_title: null

Meta-review: metareview: The paper addresses the pluralistic image completion problem. Initial reviews were borderline accept (2x) , weak accept (2x). The authors provided a rebuttal. Both borderline reviewers upgraded to weak accept. The AC agrees and considers the paper a solid contribution to NeurIPS and recommends acceptance.
confidence: Certain
Decision: Accept