Pluralistic Image Completion with Gaussian Mixture Models
Authors: Xiaobo Xia
Wenhao Yang
Jie Ren
Yewen Li
Yibing Zhan
Bo Han
Tongliang Liu
Venue: NeurIPS 2022 Accept (2022)

Abstract: Pluralistic image completion focuses on generating both visually realistic and diverse results for image completion. Prior methods enjoy the empirical successes of this task. However, their used constraints for pluralistic image completion are argued to be not well interpretable and unsatisfactory from two aspects. First, the constraints for visual reality can be weakly correlated to the objective of image completion or even redundant. Second, the constraints for diversity are designed to be task-agnostic, which causes the constraints to not work well. In this paper, to address the issues, we propose an end-to-end probabilistic method. Specifically, we introduce a unified probabilistic graph model that represents the complex interactions in image completion. The entire procedure of image completion is then mathematically divided into several sub-procedures, which helps efficient enforcement of constraints. The sub-procedure directly related to pluralistic results is identified, where the interaction is established by a Gaussian mixture model (GMM). The inherent parameters of GMM are task-related, which are optimized adaptively during training, while the number of its primitives can control the diversity of results conveniently. We formally establish the effectiveness of our method and demonstrate it with comprehensive experiments. The implementation
is available at https://github.com/tmllab/PICMM.

Pluralistic Image Completion with Gaussian Mixture Models
Wenhao Yang?, Bo Han 'TML Lab, University of Sydney *Nanjing University "JD Explore Academy
Xiaobo Xia'
Jie Ren
Yewen Li
Yibing Zhan
Tongliang Liu University of Edinhurgh "Hong Kong Baptist University
*Nanyang Technological University
Abstract
Pluralistic image completion focuses generating both visually realistic and diverse results for image completion. Prior methods enjoy the empirical successes of this task. However, their used constraints for pluralistic image completion are argued to be not well interpretable and unsatisfactory from two aspects. First, the constraints for visual reality can be weakly correlated the objective of image completion or even redundant. Second, the constraints for diversity are designed to be task-agnostic, which causes the constraints address the issues, we propose an end-to-end probabilistic method. Specifically, we introduce unified probabilistic graph model that represents the complex interactions in image completion. The entire procedure of image completion is then mathematically divided into several sub-procedures, which helps efficient enforcement of constraints, The sub-procedure directly related to pluralistic results is identified, where the interaction is established by a Gaussian mixture model (GMM). The inherent parameters of GMM are task-related, which are optimized adaptively during training, while the number of its primitives can control the diversity of results conveniently. We formally establish the effectiveness of our method and demonstrate it with comprehensive experiments. The implementation is available at https://gghubb.commmmlaaPPPlo
work well. In this paper,
Introduction
Pluralistic image completion refers to the task of filling in the missing region of an incomplete image, so to produce visually realistic and diverse image completion solutions [59-61]. Different from single image completion [39, 12, 55, 50] that learns deterministic mapping from an incomplete image to a complete image, and produces a unique result, pluralistic image completion can generate various results with visually realistic contents. Pluralistic image completion follows the fact that image completion is highly subjective process and benefits a series of applications such as photo restoration [27], object removal [13], and transmission error concealment [37].
Pluralistic image completion is complicated and challenging. The state-of-the methods encode images to latent features. To constrain the visual reality of results, heuristic constraints are intro- duced, e.g., the perceptual loss [27] and attention module [59, 44]. To constrain the diversity of results, the distribution of latent features is assumed to be unimodal Gaussian distribution with predefined parameters [59]. Image completion is performed by sampling from the unimodal Gaussian distribution.
Although the above paradigm enjoys empirical successes of pluralistic image completion, is well interpretable and unsatisfactory in practice. We detail the issues from two aspects. First, the
"The first two authors made equal contributions. t Corresponding author: Tongliang Liu (tongliang, liu @sydncy.cdu.au).
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

added constraints for the visual reality of results are based on general purposes. These constraints may work well in some tasks. Unfortunately, there is no clear understanding of what role they play in pluralistic image completion. Inappropriate constraints are likely to have side effects. It is not easy to finish the determination of added constraints as our desideratum, especially in complex image completion tasks. Second, the diversity of image completion results is hard to be constrained in reason. For a specific task, the parameters the unimodal Gaussian distribution are directly related to the diversity. Nevertheless, these parameters are designed to be prior knowledge and task-agnostic, which cause unreasonable pluralistic results.
In this paper, to address the above is- sues, we present an end-to-end prob- abilistic method. The method is con- structed with a unified probabilistic graph model (PGM) [17, 22] and translates the problem of pluralistic image completion into structured mathematical representation. Specif- ically, we suppose that pluralistic im- age completion can be represented with a directed acyclic graph that is designed reasonably. The nodes of the graph denote different kinds of images and corresponding latent fea- tures. The edges of the graph repre- sent the interactions between nodes. Based on this directed acyclic graph, the entire procedure of pluralistic im- age completion is divided into several sub-pn uub-procedures. Then, we perform different constraints for different sub- procedures as our desideratum.
(a) Input
(b) PIC
(c) Ours
Figure 1: Example completion results of the proposed method on the images of faces and natural sceneries with various masks (missing regions are shown in gray). The baseline PIC refers [59]. For each group, the masked input image is shown left, followed by diverse and plausible com- pletion results from our method without any post-processing. (Zoom in to see the details.)
In particular, the diversity of image completion results is implemented by modeling the interaction between the latent features of the missing region and incomplete image with a Gaus- sian mixture model (GMM) [40, 64).
Compared with the unimodal Gaussian distribution, GMM has larger capacity and is more com- petitive to meet the output diversity [56, 38]. For our method, the inherent parameters of GMM are task-related and optimized adaptively during training. The different primitives of GMM represent the outputs with different patterns. Additionally, the number of primitives can be chosen artificially with the needs for diversity in real- world applications.
Before delving into details, we clearly emphasize our contributions as follows:
To best knowledge, this paper is the first that uses PGM for pluralistic image completion. Besides, the PGM is specially designed with justifications, rather than borrowing existing models. With the designed PGM, the added constraints for visual reality and diversity are explainable.
We propose to use GMM to increase result diversity. One remarkable advantage is that the inherent parameters for diversity are task-related, leading more reasonable results.
We conduct series of experiments on benchmark datasets support our claims. In both qualitative and quantitative comparisons with the state-of-the-art methods, our method achieves superior performance. The generated contents for image completion are both visually realistic and diverse, such as those shown in Figure 1.
The rest of the paper is organized as follows. In Section 2, we briefly review pluralistic image completion. Section 3, we discuss the proposed method step by step. Experimental results are provided in Section 4. Finally, we conclude the paper in Section 5.

---

2 Preliminaries
Problem Setting. We first discuss image completion. For the task of image comple- tion, suppose we have an image, originally Io. but degraded by some missing pixels to become masked partial image 1n- The image that comprises the original missing pixels is called complement partial image, which can be denoted by I.- The goal the image completion task is to reconstruct I, by using Im- The illustrations for different kinds of images are provided in Figure 2. Traditional single image completion reconstructs the original image in deterministic fashion, which results in only a single solution [12, 9].
Ig
L
Figure 2: The illustrations for different kinds of images in image completion.
Pluralistic image completion [59] is motivated by the fact that image completion is highly subjective process. Also, completed images by different experts may agree on high-level semantics, but have need to generate multiple and diverse plausible results when presented with a masked partial image. The problem of pluralistic image completion is challenging since need to take of the diversity and visual authenticity of results at the same time.
substantially different details [59]. Therefore, in pluralistic image completion,
Prior Work. Traditional methods on single image completion, e.g. diffusion-basee methods [3, 1, 23] and patch-based methods [2, 4, 5], assume that image holes share similar content to visible regions. Therefore, they choose to directly propagate the contextual appearances or realign the background patches to complete the image holes. Benefiting from the strong power of deep neural networks, deep learning based single image completion methods achieve promising completion performance, egg., employing shepard convolutional neural networks [39], restraining global and local consistency between image holes and visible regions [16]. performing deep feature rearrangement [47], and learning pyramid-conter encoder network [53], etc. In addition to these methods, multiple single image completion methods involve the generative adversarial networks [10] learn the semantic of images [48, 49], e.g., e. [25, 51, 8, 6, 57, 12, 9]. Although these single-solution methods achieve outstanding performance in predicting deterministic result for image holes, they cannot generate various semantically meaningful results.
Existing works for pluralistic image completion usually rely generative models [24], e.g., CVAE [43, 13, 591, hicrarchical VQ-VAE [37], PD-GAN [27], and BicycleGAN [63]. Generative pluralistic image completion methods can diversify meaningful completion results with different kinds of constraints. However, such methods are not well interpretable as discussed. The issues largely limit their applications in the real world.
3 Methodology
This section presents a probabilistic method for pluralistic image completion. We first design probabilistic graph model for our task (Section 3.1). Then, based on the probabilistic graph model, we show how to divide the entire procedure of pluralistic image completion into several sub-procedures (Section 3.2). Afterward, we discuss how to use GMM to diversify image completion results (Section 3.3). Finally, we add reconstruction and adversarial losses to strengthen image completion, and summarize all algorithm flows (Section 3.4).
3.1 Probabilistic Graph Model Construction
Recall the notations in image completion, i.e., Ic, I,, and I., they correspond to the latent features Zo. Zm, and z respectively. We suppose a probabilistic graph model for the image completion task. which is shown in Figure 3. The probabilistic graph model is designed reasonably. The graphical illustration of overall computational paths of the probabilistic graph model is presented in Figure 3(d). Specifically, the images Im and I, form I,. Their latent features 2m and 2, can be exploited to form I,. Given 2m, we can surmise Z.. We use the latent features here, since they are low-dimensionall and informative [11, 21, 45, 46]. We further details Figure 3 as follows.

---

Zc
(a) Inference
(b) Dynamic Inference
(c)
Generation
(d) Overall
Figure 3: The illustrations of the probabilistic graph model for the pluralistic image completion task.
Inference. We do not directly map I. and I to I.. since they are too high-dimensional and have information redundancy. We exploit generative models. The processes Im - Zm and I. - Ze denote that we encode images into corresponding latent features. During training, we learn the encoder ff to finish the inference.
Dynamic Inference. Given Im (resp. I,), we can infer the latent feature (resp. ze). Afterward, Zm --> Ze means that we use variational distribution p(zelzm) to approximate the distribution p(z. Here we denote this relationship using dashed line, which is designed to be dynamic to diversify outputs.
Generation. image completion, the images Im and I can compose the image I,. Accordingly, for their latent features, splice Zm and be output I, During training, we learn the decoder g finish the generation.
3.2 Image Completion Objective Decomposition
We cannot observe the underlying in the test procedure. Based the probabilistic graph model, we perform the following variational inference with the Kullback-Leibler (KI.) divergence:
Lp KL [ga (lo, Zm Ze Im >lpp 2m, Ze Am Ic)]
(1)
where qo (-I-) and (-1-) are two conditioned distributions, with t and o being the parameters
of their corresponding functions. The divergence (1) is minimized with respect all parameters. It should be noted that, in fact, the overall objective should be KL(g0(I) (Io [o]lm))lp (I. I.)] = [L. (I Zm, Ze [I (po (T. Zm, [))]l- As the expectation in this formula is intractable, we need to deal with the 1-step Monte Carlo sampling objective, which is why the (1) that constraining each sampling point can make the overall constraint tighter. We further decompose the divergence (1)
divergence (1) is formed. Another advantage of using the divergence
based on the probabilistic graph model follows.
Proposition Regarding the KI. divergence (I), we show that the divergence can decomposed as:
Lp KL (g (Io, Zm Ze [1 Po (I,, Zm, Ze Im, Ic)
Ee :2, Jrigie mm KL a (To Vem, 2o) pl (Io) (Io Am, Ic)!
+ Eran mo (Zim (T..)) KL 190 (ze Zm (l Ic)] + KL [qw Imm 1l pol Zm [m)),
where qo is the variational distribution, and the parameters are involved in divergence minimization.
The proof can be found in Appendix A.I. We analyze three terms in Eq. (2) combining the proba- bilistic graph model and our purpose:
For @. as discussed, we can generate I, with I,, and L., with their latent features and Ze- We minimize the distance about the generation I, in the two ways. The minimization can ensure that the generation from the images and latent features is consistent.
For b, we do not limit the inference from I, to 2, deterministic function. Instead, to diversify outputs, we use the variational distribution to restrain go (2..l..))) The choice and discussion about the variational distribution will be provided in Section 3.3.

---

For ( it aims to keep that the inference using the variational distribution and the inference using the posterior close, which also guarantees the reliability of sampling. Besides, the variational distribution does not include I, to enable latent features to be used in test. We employ the variational evidence lower bound (ELBO) TELBO [21, 15] to constrain C because, suppose that p( a constant given I.. it is equivalent for maximizing the LELBO and minimizing @ The details are provided in Appendix A.4.
3.3 Gaussian Mixture Model
As discussed, we need a suitable choice for the variational distribution go go Zm)) to output pluralistic results. In this work, we employ the Gaussian Mixture Model (GMM) [30. 31] for the dynamic mode qe (z.z)/2 Zm,), which is more flexible than a unimodal and can diversify outputs.
We denote the latent features of I that inferred from Zm by 2c. which is distinguished from z. achieved by the encoder. For the latent features z, we denote the mean and covariance by jaM and Σ² Mathematically, we have
qo (Ze Zm)
Σ aw (Zol ) i=1
where &t denotes the number of primitives, Or denotes the mixing coefficient of the i-th primitive, and 14, E, denotes the distribution parameters of the i-th primitive. Note that the mixing coefficient o; is the parameter of the Categorical distribution [31]. Intuitively, in Eq. (3), we model Zm with GMM, where the i-th primitive can be used represent (8) In this way, diversity requirements.
obtain different 2. to meet
Besides, among all the parameters of GMM, except for k. the other parameters are called the inherent parameters of GMM. In GMM (3), the number of primitives can determined artificially according to the need for output diversity. The inherent parameters optimized adaptively during training. Specifically, for the optimization of the mixing coefficients a [ai...., an). we exploit the the frequency loss [38]. The objective is
LF = (v a) a) v [ and
1,j argmin KL [N(Zoll D IIN M
0, otherwise.
For our task, with the frequency loss (4), the accumulated frequency approximate gradient is an asymptotically unbiased estimation of the true gradient. We provide the detailed derivation and analysis Appendix A.2. The reason we mention the frequency loss is that, the metric for evaluating the performance of each primitive is adapted to KL divergence in our task, which is different from the original metric in [38]. For the optimization of μ and Σ of Zmz we utilize the back-propogate- max-operation to reserve the distinguishable property of GMM (3). For our problem, the objective of the back procokattpgateeeeee maxeeeemaa-aaae wax-mmax---aaeeaaee max-opeeatiion is shown in the following proposition.
Proposition 2
By modeling the variational distribution with GMM (3), we have
1 tr
1 (1 1 ²)) )
LBM log
- '1 1²²))
(5)
where j argmin, KL N(zc/u E )1 } and tr(-) denotes the trace of matrix.
The derivation of Proposition can be found in Appendix A.3. For Proposition 2, if we minimize LEM, optimize a and Σ of z,.. which reduces the distance between 20))) and z.. For the term b LBM-
in Eq. (2), we then have LCMM = Ez.,. ~ag (#... IT..) KL [go zm (Z. = L+
Discussion. Note that the diversity of results is directly related to Zm When GMM for z,, is dominated by the i-th primitive (a; &s 1), the diversity of image completion results reduced. Although this phenomenon did not appear in our experiments, it may exist in practice. Actually, this phenomenon is reasonable. From human cognition, if Im shows only one pattern, the inference for I. will be deterministic [3]. For example, if the size of the missing region of incomplete image is
5

---

very small, we can infer the missing region both deterministicalyy and reasonably, where the diversity of results is reduced. Prior methods with task agnostic constraints do not consider this fact, which may create unreasonable results. For our method, the constraints for diversity are task-related, which can take this phenomenon into account.
Algorithm 2 Test procedure
Algorithm 1 Training procedure
Input: the image Im. the num- ber of primitives of GMM k, the trained encoder f", and decoder 9".
Input: images Lo, Im and Lc, the number of primitives of GMM k the initialized encoder f. and decoder g.
1: Encode Im (resp. Ig) to Zm (resp. 2.)) with S:
2: Model Zm with GMM and ze with a unimodal Gaus-
Encode Lm Zm with f";
sian distribution:
2:
Model Zm with GMM;
3: Calculate the loss LOMM as discussed in Section 3.3;
for j= do
4: Update the parameters of GMM with LGMM;
4: Sample i with
5: Infer 2()) from Zm with Eq. (3) and Eq. (4),
Categorical(q1... ax);
I....,k;
5: Infer 2 with 2.6)
6: Generate images by with I, = g(Zm, Zo) and
N( pm ETT):
160 = g(Zm. = 2c 2(2)), j = L.....,;,
6: Generate images 1 by g" with
7: Calculate the loss LELBO and Le discussed in Sec-
1 =g"(zm 2
tion 3.2 and 3.4;
7: end for
8: Update all the parameters w.E.. LELBO and Lc.
Output: pluralistic image completion sults )
Output: the trained encoder f* and decoder g"
3.4 Reconstruction and Adversarial Losses
To ensure the minimization of the term @ in Eq. (2), we incorporate the use of reconstruction and adversarial losses [10]. The reconstruction loss is formulated
LR
E li..
1ol1 + E
1 [mlla
(6)
where I, is generated from Zm and Ze. and 1 is generated from 2 Here, ( can be inferred from Zm by using Eq. (3) and Eq. (4). The image 1 is obtained from i!!) with the image mask that can degenerate I, to Im- The image i is generated from Zm and 2 The reconstruction loss (6) controls the visual rationality of image completion results. Also, we efficiently avoid the deterministic fashion with GMM. Furthermore, we leverage adversarial training [10] to make the generated images more realistic. The adversarial loss LA formulated
L EE [(t)) 11 + E [ID((T") - 1(1.)
where D is the diseriminator optimized by the discriminator loss based on LSGAN [29]. We jointly train our encoder of and decoder g through the following combined loss:
Le = LR + AALA.
(8)
where the weight X is set to 0.05 in all experiments. Then, the final objective of the loss function is the sum of three losses, i.e., LFINAL. = LGMM + LELBO + Le. The final loss is to maximize the log-likelihood of the conditional data distribution. Specifically, the adversarial loss Wasserstein distance to make generated images realistic given latent features. The ELBO loss uses KI. divergence and distribution likelihooo make variational posterior approximate real posterior.
the
Algorithm Flows. For the convenience of following technical details, we provide the algorithm flows. The algorithm flows of training and test stages can be found in Algorithm and Algorithm 2. The illustration is shown Appendix B.
It should be noted that the number of primitives & does not mean that we are limited to generate only l diverse results given I, (in Steps 3 and 4 of Algorithm 2). fact, we can sample from the primitives to obtain lots of image completion results. Moreover, as GMM has much larger capacity than the unimodal Gaussian distribution [31], we can achieve greater diversity. We provide empirical observations in Section 4.3 and Appendix C.

---

Experiments
In this section, we conduct series of experiments to justify our claims. We first introduce the implementation our method (Section 4.1). The comprehensive experimental results and comparison with advanced methods are then provided and discussed (Section 4.2). Finally, we conduct an analysis study to present and discuss our method in more detail (Section 4.3).
4.1 Implementation Details
Datasets. We evaluated our proposed model on five popularly used datasets, i.e., CelebA-HQ [18. 28], FFHQ [19], Paris StreetView [7], Places2 [62], and ImageNet [41]. We verify the effectivness of our method with different types of mask regions, including both center and random masks [59, 44].
Network and Optimization. The proposed method can be implemented efficiently. The encoder and decoder of our pipeline inspired by PIC [59] for comparison. We apply average pooling and interpolation with convolutional layers to implement downsampling and upsampling respectively. During optimization, use the Adam optimizer [20]. The learning rate is fixed to 10-1 during the training procedure.
Baselines. We compare the proposed method with the following state-of-the-arr methods, which include (1) single image completion methods: DFv2 [52]. EC [33], and MED [26]; (2) pluralistic image completion methods: PIC [59] and ICT [44]. We abbreviate our method (Pluralistic Image Completion with Gaussian Mixture Models) as PICMM. The methods are implemented by PyTorch and evaluated NVIDIA Tesla A100 GPUs.
(a) Input
(b) PIC
(c) Ours
(a) Input
(b) EC
(c) ICT
(d) Ours
Figure 4: Qualitative comparison of our method with PIC on CelebA-HQ (first 4 rows) and FFHQ (last 2 rows). Best viewed by zooming in.
Figure Qualitative comparison of our method
with EC and ICT on Places2. Best viewed by zooming
(a) Input
(c)
(d)
(a) Input
(b)
(c) Ours
Figure 5: Qualitative comparison of our method with MED and PIC on Paris StreetView. Best viewed by zooming in.
Figure 7: Qualitative comparison of our method
with ICT on ImageNet. Best viewed by zooming in.
Measurement. For measurement, we provide both qualitative and quantitative results. For the metrics of quantitative results, common evaluation metrics such as the peak signal-to-noise
7

---

ratio (PSNR), structural similarity (SSIM), mean absolute error (MAE), and fréchet inception dis- tance (FID) [14] to measure the similarity between the image completion result and ground truth. Furthermore, pluralistic image completion is supposed to focus on generating diverse realistic results rather than merely approximating ground-truth ones [58]. To measure the result diversity, we add two perceptual quality metrics quantitative results, which include LPIPS [59] and DIV-FID [44].
4.2 Comparison with Prior Methods
4.2.1
Qualitative Comparisons
We provide extensive qualitative comparison re- sults to justify our claims. First, we show the results on CelebA-HQ and FFHQ in Figure 4, which are images related human faces. can be seen, the image completion results achieved by our method are realistic, following high im- age quality. Also, compared with the baseline PIC, our completion results are more diverse, e.g., the images in the second and last rows.
Second, we provide the results on Paris StreetView and Places in Figures 5 and 6 re- spectively. For the results on Paris Street we can see that image completion by the base- lines MED and PIC are somewhat distorted. By contrast, our method can finish the image com- pletion task better. The details can be checked the complements to walls. Then, we turn the attention the image completion results Places2. Our method still achieves superior performance compared with baselines.
Figure 8: Refined result analysis of our method. The face images come from CelebA-HQ.
Lastly, we provide the results on ImageNet in Figure 7. Note that the sources of the data in ImageNet are very complicated. The image completion task rather challenging on this dataset, even for single image completion [59]. Different from some prior work [36, 16] that were trained on 100k subset of training images of ImageNet, we directly the network on the original ImageNet training dataset with all images. The results mean that our method can infer the content effectively.
refined results in Figure 8 analyze the image comple- tion details of our method. For the face images, we can find that the image completion results have very different facial expressions.
Refined Result Analysis. We provide
Dataset
FFH
Places2
Method
Mask
PSNR 1
SSIM 1
MAE
FID 4
PSNR
SSIM
MAE $
FID +
DFv2 [52]
25.868
0.922
0.0231
16.278
26.533
0.881
0.0215
24.763
EC [33]
26.901
0.938
0.0209
14.276
26.520
0.880
0.0220
25.642
MED [26]
26.325
0.922
0.0230
14.791
26.469
0.877
0.0224
26.977
Center
PIC [59]
26.781
0,933
0.0215
14.513
26.099
0.865
0.0236
26.393
ICT [44]
27.922
0.948
0.0208
10.995
26.503
0.880
0.0244
21.598
PICMM
27.551
0.937
0.0203
10.604
26.554
0.886
0.0208
24.373
DFv2 [52]
24.962
0.882
0,0310
19,506
25,692
0.834
0.0280
29.981
EC [33]
25,908
0.882
0.0301
17.039
25.510
0.831
0.0293
30.130
MED [26]
25.118
0.867
0.0349
19.644
25.632
0.827
0.0291
31.395
Random
PIC [59]
25.580
0.889
0.0303
17.364
25.035
0.806
0.0315
33.472
ICT [44]
26.681
0.910
0.0292
14.529
25.788
0.832
0.0267
25.420
PICMM
25.950
0.912
0.0289
17.014
25.310
0.829
0.0236
25.025
Table I: Quantitative results on FFHQ and Places2 datasets with different mask settings. The best
results are in bold.
4.2.2 Quantitative Comparisons

---

Common Metrics, Quantitative eval- uation with common metrics diffi- cult for pluralistic image completion, as our goal is to obtain diverse but reasonable solutions for masked im- age [59]. To make a feasible compar- ison, did in [59], we sample one completed image many times from images to compare with the original image. The results are provided in Tables 1 and 2. As can be seen, for FFHQ and Places2, on both the set- tings of center and random masking, method achieves superior performance. On ImageNet, our method performs the best with the center masking consistently, while it is competitive with the random masking.
Dataset
ImageNet
Method
Mask
PSNR 1
SSIM 1
MAE 1
FID
PIC [59]
24.010
0.867
0,0319
47.750
ICT [44]
Center
24.757
0.888
0.0263
28,818
PICMM
24.932
0.897
0.0260
23.718
PIC [59]
22.711
0.791
0.0462
59.428
ICT [44]
Random
23.775
0.835
0.0358
35.842
PICMM
23.322
0.846
0.0415
39.742
Table 2: Quantitative results on ImageNet with different
mask settings. The best results are in bold.
should be noted that ICT is a very strong baseline and works better than our method in some cases. It because ICT is built with the image transformer [35], which is powerful in vision tasks. Compared with ICT, the advantages of our method lies not only better interpretability as discussed, but also faster inference speeds. We will demonstrate this in inference time.
Diversity Metrics. For pluralistic im- age completion, the diversity evalua- tions are significant. We generate five completion images in each task and report the mean the diversity met- ric. The results are shown in Table 3, which represents that our method can generate more diverse completion im- ages than baselines. The results with common and diversity metrics support our claim very well. That is to say, the image completion results of our method are both high-quality and pluralistic.
Dataset Method
FFH
Places2
LPIPS
DIV-FID 1
LPIPS
DIV-FID f
PIC[59]
0.029
9.130
0.047
17.742
ICT [44]
0.065
13,909
0.089
25.253
PICMM'
0.071
17.361
0.092
27.194
Table 3: The diversity result comparison on FFHQ and
Places2 datasets with the center mask setting. The best re- sults are in bold.
Inference Time. We report the inference time of our method, compared with PIC and ICT. All
Method
Inference time (minutes) 4
inference runs on one NVIDIA Tesla A100 GPU
PIC [59] ICT [44]
0.387
for fairness. The size of images is 256 x 256.
179. 179.126
perform pluralistic image completion for 100
PICMM
0.235
different images. For each image, generate 6 completion results for We report the total infer- ence time for these images. The results shown in Table 4. Note that as ICT is transformer-based method, the calculated consumption of its infer- ence is much heavier. What's worse, ICT needs iterative Gibbs sampling during inference. The two issues make ICT have face heavy computational consumption, which is also mentioned [44]. In contrast, our method is more inference-efffcienee which is more than 750 times faster than ICT. The merit could make our method easier use in the real world.
Table 'he inference time comparison with PIC and ICT in the center mask setting. The best result is in bold.
4.3 More Analyses and Justifications
GMM Primitive Visualization. We further stress the diversity of the completion results achieved by our method. Specifically, we visualize the 2. which inferred from Zm as discussed. The visualization results are obtained with t-SNE [42]. Different 2, presented by the 2D-vectors in different colors, which are shown Figure 9. As can be seen, the vectors are scattered, which clearly demonstrates that the completion results of our method are diverse.
Refined Diversity Comparison. We argue that, benefiting from that GMM has larger capacities than a unimodal Gaussian distribution, the completion results of our method would be more diverse. To justify our claims, we provide the refined diversity comparison our method with PIC and ICT, as shown in Figure 10, Note that the six image completion results of PIC are obtained by sampling from

---

Figure 9: GMM primitive visualization plu- ralistic completion results by method. The face image comes from CelebA-HQ. The scene image comes from Places2.
Figure 10: Refined diversity comparison with PIC (the first row) and ICT (the second row). The original image in this figure comes from CelebA-HQ.
unimodal Gaussian distribution. The results of our method are obtained by sampling from six primitives. Clcarly, our results are more diverse than PIC's. Besides, for ICT, it does not explicitly control the output diversity. The diversity of pluralistic image completion is thus less interpretable than our method.
In addition, we state that our method is not limited to only generating k completion results before. In fact, we can sample from primitives of GMM to obtain lots of completion results. Due to the limited page, we present the results and more analyses in Appendix C. Moreover, the richer completion result comparison presented Appendix
Conclusion
In this paper, we focus on the complicated and challenging problem of pluralistic image completion. We propose novel end-to-end probabilistic method for this challenging problem. Based on the probabilistic graph model, our method divides the entire procedure of pluralistic image completion into several sub-pu uuo-procedures, where GMM used diversify outputs. Experiments on variety of datasets show that the image completion results by our method are both high-quality and pluralistic. In future work, investigating the feasibility of the proposed method for other diverse decision-makinn scenarios might prove important.
Acknowledgements
Yibing Zhan was partially supported by the Major Science and Technology Innovation 2030 "New Generation Artificial Intelligence" key project (No. 00212D0111700 and the National Natural Science Foundation of China (Grant No. 62002090). Xiaobo Xia was supported by Australian Research Council Projects DE- 10-11000143 and Google PhD Fellowship. Bo Han was supported by the RGC Early Career Scheme No. 22200720, NSFC Young Scientists Fund No. 62006202, and Guangdong Basic and Applied Basic Research Foundation No. 2022A151501 1652. Tongliang Liu was partially supported by Australian Research Council Projects DP180103424, DE- 09010101,4 IC- 190100031, DP- DP-220102121, and FT-2201003 8. The authors would give special thanks to Mingrui Zhu (Xidian University). Zihan Ding (Princeton University), and Chenlai Qian (Southeast University) for helpful discussions.
References
[1] Coloma Ballester, Marcelo Bertalmio, Vicent Caselles, Guillermo Sapiro, and Joan Verdera. Filling-in by joint interpolation of vector fields and gray levels. IEEE Transactions on Image Processing, 10(8): (180:1200-1211 2001.
[2] Connelly Baress, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. Patchmatch: A randomized correspondence algorithm for structural image editing. ACM Transactions on Graphics, 28(3):24, 2009.
[3] Marcelo Bertalmio, Guillermo Sapiro, Vincent Caselles, and Coloma Ballester. Image inpainting. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pages 417-424, 2000.
10

---

[4] Marcelo Bertalmio, Luminita Vese, Guillermo Sapiro, and Stanley Osher. Simultaneous structure and texture image inpainting. IEEE Transactions on Image Processing, 12(8):882-88) 889, 2003,
[5] Antonio Criminisi, Patrick Pérez, and Kentaro Toyama. Region filling and object removal by exemplar-based image inpainting. IEEE Transactions on Image Processing, 13(9):1200-122 1212. 2004.
[6] Jiankang Deng, Shiyang Cheng, Niannan Xue, Yuxiang Zhou, and Stefanos Zafeiriou. Uv-gan: Adversarial facial uv map completion for pose-invariant face recognition. In CVPR, pages 7093-7102, 2018.
[7] Carl Doersch, Saurabh Singh, Abhinav Gupta, Josef Sivic, and Alexei Efros, What makes paris look like paris? ACM Transactions on Graphics, (4)) 2012.
[8] Brian Dolhansky and Cristian Canton Ferrer. Eye in- in-pannting with exemplar generative adver- sarial networks. In CVPR. pages 7902-7911, 2018.
[9] Haoye Dong, Xiaodan Liang, Yixuan Zhang, Xujie Zhang, Xiaohui Shen, Zhenyu Xie, Bowen Wu, and Jian Yin. Fashion editing with adversarial parsing learning. In CVPR, pages 8120-8122111 2020.
[10] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS, 2014.
[11] Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1.
MIT Press, 2016.
[12] Artur Grigorev, Artem Sevastopolsky, Alexander Vakhitov, and Victor Lempitsky. Coordinate- based texture inpainting for pose-guided human image generation. In CVPR, 2019.
[13] Xintong Han, Zuxuan Wu, Weilin Huang, Matthew R Scott, and Larry S Davis. Finet: Compati- ble and diverse fashion image inpainting. In ICCV, pages 4481-4491, 2019.
[14] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to local nash equilibrium. In NeurIPS, 2017.
[15] Zhuo Huang, Xiaobo Xia, Li Shen, Bo Han, Mingming Gong, Chen Gong, and Tongliang Liu. Harnessing ottoo-ddistriution examples via augmenting content and style. arXiv preprint 27X.v:220..03162, 2022.
[16] Satoshi lizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. Globally and locally consistent image completion. ACM Transactions Graphics, 36(4):1-14, 2017.
[17] Michael Jordan. An introduction to probabilistic graphical models, 2003.
[18] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv: 2017.
[19] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, pages 4401-4410, 2019.
[20] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In JCLR, 2015.
[21] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint 131266 14, 2013.
[22] Daphne Koller and Nir Friedman. Probabilistic Graphical Models: Principles and Techniques. MIT press, 2009.
[23] Anat Levin, Assaf Zomet, and Yair Weiss. Learning how inpaint from global image statistics. In ICCV, pages 305-312, 2003.

---

[24] Yewen Li, Chaojie Wang. Xiaobo Xia, Tongliang Liu, Xin Miao, and Bo An. cut-of-distribution detection with an adaptive likelihood ratio on informative hierarchical vae. In NeurIPS, 2022..
[25] Yijun Li, Sifei Liu, Jimei Yang, and Ming- -uugn Yang. Generative face completion. In CVPR, pages 3911-3999 1-3919, 2017.
[26] Hongyu Liu, Bin Jiang, Yibing Song, Wei Huang. and Chao Yang. Rethinking image inpainting via a mutual encoder-decoder with feature equalizations. In ECCV, pages 721-741, 2020.
[27] Hongyu Liu, Ziyu Wan, Wei Huang, Yibing Song, Xintong Han, and Jing Liao. Pd-gan: Probabilistic diverse gan for image inpainting. CVPR, pages 9371-9381 2021.
[28] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang, Deep learning face attributes in the wild. In ICCV, pages 3730-3738, 2015.
[29] Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley, Least squares generative adversarial networks. In CVPR, pages 2794-2802, 2017.
[30] Geoffrey J McLachlan and Suren Rathnayake. On the number of components in gaussian mixture model. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 4(5): 341-355, 2014.
[31] Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of Machine Learning. MIT press, 2018.
[32] Daniel Mas Montserrat, Hanxiang Hao, Sri K Yarlagadda, Sriram Baireddy, Ruiting Shao, János Horváth, Emily Bartusiak, Justin Yang. David Guera, Fengqing Zhu, et al. Deepfakes detection with automatic face weighting. CVPR Workshops, pages 668-669, 2020.
[33] Kamyar Nazeri, Eric Ng. Tony Joseph, Faisal Qureshi, and Mchran Ebrahimi. Edgeconnect: Generative image inpainting with adversarial edge learning. arXiv preprint 1XXi:1901.00222 2019.
[34] Thanh Thi Nguyen, Cuong M Nguyen, Dung Tien Nguyen, Duc Thanh Nguyen, and Saeid Na- havandi. Deep learning for deepfakes creation and detection. arXiv preprint 1rXiv:I909.111 11573, 2019.
[35] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In ICML, pages
2018.
[36] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexci A Efros. Context encoders: Feature learning by inpainting. In CVPR, pages 2536-2544, 2016.
[37] Jialun Peng, Dong Liu, Songcen Xu, and Houqiang Li. Generating diverse structure for image inpainting with hierarchical va- In CVPR, pages 10775-10784, 2021.
[38] Jie Ren, Yewen Li, Zihan Ding, Wei Pan, and Hao Dong. Probabilistic mixture-of-expett for efficient deep reinforcement learning. arXiv preprint arXiv:2104.09122, 2021.
[39] Jimmy S Ren, Li Xu, Qiong Yan, and Wenxiu Sun. Shepard convolutional neural networks, In NeurlPS, 2015.
[40] Douglas A Reynolds. Gaussian mixture models. Encyclopedia of Biometrics, 741:659-663, 2009.
[41] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211-552, 2015.
[42] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(11), 2008
[43] Jacob Walker, Carl Doersch, Abhinav Gupta, and Martial Hebert. An uncertain future: Fore- casting from static images using variational autoencoders. In ECCV, pages 835-851, 2016.
12

---

[44] Ziyu Wan, Jingbo Zhang, Dongdong Chen, and Jing Liao. completion with transformers. In ICCV, 2021.
pluralistic image
[45] Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang. Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Part-dependent label noise: Towards instance-dependent label noise. NeurIPS, 2020.
[46] Xiaobo Xia, Bo Han, Nannan Wang, Jiankang Deng, Jiatong Li, Yinian Mao, and Tongliang Liu. Extended T: Learning with mixed closed-set and open-set noisy labels. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[47] Zhaoyi Yan, Xiaoming Li, Mu Li, Wangmeng Zuo, and Shiguang Shan. Shift-net: Image inpainting via deep feature rearrangement. In ECCV, pages 1-17, 2018.
[48] Shuo Yang, Wei Yu, Ying Zheng, Hongxun Yao, and Tao Mei. Adaptive semantic-viste tree for hierarchical embeddings. In ACM MM, pages 2097-2105, 2019.
[49] Shuo Yang, Peize Sun, Yi Jiang, Xiaobo Xia, Ruiheng Zhang, Zehuan Yuan, Changhu Wang, Ping Luo, and Min Xu. Objects in semantic topology. In ICLR, 2022.
[50] Zili Yi, Qiang Tang. Shekoofeh Azizi, Daesik Jang, and Zhan Xu. Contextual residual aggrega- tion for ultra high-resolution image inpainting. In CVPR, pages 7508-7517, 2020.
[51] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Generative image inpainting with contextual attention. In CVPR, pages 5505-5514, 2018.
[52] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Free-form image inpainting with gated convolution. In ICCV, pages 4471 4471-440 1470,, 2019.
[53] Yanhong Zeng, Jianlong Fu, Hongyang Chao, and Baining Guo. Learning gyramid-conter encoder network for high-quality image inpainting. In CVPR, pages 1486 1494, 2019.
[54] Yanhong Zeng, Jianlong Fu, Hongyang Chao, and Baining Guo. Aggregated contextual transformations for high-resolution image inpainting. IEEE Transactions on Visualization and Computer Graphics, 2022.
[55] Yu Zeng, Zhe Lin, Huchuan Lu, and Vishal M Patel. Cr-fill: Generative image inpainting with auxiliary contextual reconstruction. In ICCV, pages 14164-14173, 2021.
[56] Haizheng Zhang, Lee Giles, Henry C Foley, and John Yen. Probabilistic community discovery using hierarchical latent gaussian mixture model. In AAAI, pages 663-668 668, 2007.
[57] Haoran Zhang, Zhenzhen Hu, Changzhi Luo, Wangmeng Zuo, and Meng Wang. Semantic image inpainting with progressive generative networks. In ACM MM, pages
2018.
(58) Lei Zhao, Qihang Mo, Sihuan Lin, Zhizhong Wang. Zhiwen Zuo, Haibo Chen, Wei Xing, and Dongming Lu. Uctgan: Diverse image inpainting based on unsupervised cross-space translation. In CVPR, pages 5741-5750, 2020.
[59] Chuanxia Zheng, Tat-Jen Cham, and Jianfei Cai. Pluralistic image completion. In CVPR, pages 1838-1447,400 2019.
[60] Chuanxia Zheng, Tat-Jen Cham, and Jianfei Cai. Pluralistic free-form image completion. International Journal of Computer Vision, 129(10) 270022 2021.
[61] Chuanxia Zheng, Tat-Jen Cham, and Jianfei Cai. Thill: Image completion via a transformer- based architecture. arXiv preprint 104v:22004008445 2021.
[62] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(6):1452-1466 2017.
[63] Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A Efros, Oliver Wang, and Eli Shechtman. Multimodal image-to-image translation by enforeing bi-cycle consistency. In NeurIPS, pages 465-476, 2017.
13

---

[64] Bo Zong, Qi Song, Martin Renqiang Min. Wei Cheng, Cristian Lumezanu, Daeki Cho, and Haifeng Chen. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In ICLR, 2018.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes],
(b) Did you describe the limitations of your work? [Yes];
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes].
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent obtained from people whose data you'le 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "umvhVlP9zY3", "reviewer": "Reviewer_Dmgt", "summary": "For pluralistic image completion, the paper presents a probabilistic method that introduces a Gaussian mixture model(GMM) to build the interaction of some variables in image completion procedure. The GMM whose parameters are task-related is beneficial to the diversity and reliability of completion results. The method is evaluated on five datasets, i.e., CelebA-HQ, FFHQ, Paris StreetView, Places2, and ImageNet.", "questions": "Although I think this paper is an overall interesting paper, and most the experimental results are effective, I still have some concerns about this paper (more details could be found in weakness):\n\n-the motivation  is a bit hard to follow;\n\n-some more comparison to the related works is required;\n\n-some details of evaluation are also required;\n\n-the writting also could be improved.  ", "limitations": "The authors claimed that the limitation is described in the checklist, however, I didn't find the explicit discussion about the limitation. I find some comparison to ICT, which obtains better results than the proposed method, due to the using the larger model with transformer architecture.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n+The probabilistic method is interesting, its theoretical derivation and implementation steps are easy to follow.\n\n+The proposed method is effective on the experimental datasets.\n\nWeakness:\n\n-The motivation for this paper is a bit hard to follow (in Line 5-7 and 32-41). Specifically, what is the constraints for visual reality? What does the word \"task-agnostic\" mean,  in this particular scenario, what is the specific meaning of \"task\" in this term? It's better to explain the in more details.\n\n-The paper lacks more discussions about the correlation and differences between its own method PICME and existing works for pluralistic image completion in Line 110-114. Meanwhile, the experimental tables lacks the comparison results of some cited methods, such as [39,25], etc.\n\n-About the Mixture-of-Experts Model, the size of the number of primitives k is not specifically discussed and analyzed, so the difference in effectiveness between the single Gaussian model when k=1 and the GMM when k>1 cannot be verified.\n\n-The paper has some typos and should be checked carefully, such as the lack of a predicate in Line112 and lacking quoting symbols in the experimental tables.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "RJXp37PtmD3", "reviewer": "Reviewer_Nbcm", "summary": "This paper presents a nice theoretical framework to provide a diverse set of plausible images that come from the reconstruction process under occlusion. The proposed approach aims to deal with diversity by using  a mixture of experts that are based on a mixture of Gaussians. Several comprehensive experiments seem to testify the usefulness of the proposed approach.", "questions": "- In section 3.2 it is not so clear how precisely the 1-step Monte Carlo (MC) is done. MC is a way to obtain samples from a distribution in an iterative way. Such an iterative process is not clear. The term “1-step”, refers to what, precisely …\n- Also, and related to the above and the MC question, how could it be possible to introduce the Expectation Maximization (EM) algorithm to determine the mixture components of the GMM. Or a sort of …It is well known the EM is able to estimate such components, where the means and the variances of the GMM components are iteratively updated. As it is written in the paper I can figure out the “iterative” update process. Is this done in a single one-step? Is it for this reason that the authors call the algorithm “1-step” ?    \n- Perhaps, this is the reason why I did not follow exactly the difference \\hat{z} and z_c,  … and how these samples are iteratively obtained.\nConcerning the above remark, I would recommend to include a figure regarding the architecture of the approach, to better see how the variables play in the framework. I miss the big picture here …\n- Determining the number of the mixture components (i.e. “k”) is a model selection problem. Again, taking the EM as a reference procedure, it is known that it is possible to determine the order of the model by using, for instance, the minimum description length. This issue is not addressed in the paper. So the question is how to handle this problem, that is the estimate automatically “k” in an end-to-end methodology as proposed herein.\n- Seems that all the components of (2) are modeled by using losses (I’m not sure if this is something related to the above question).  Thus:\n-term a) is modeled using L_R + \\lambda_A * L_A\n-term b) is modeled using L_GMM = LF + L_BM\n-term c) is models by using L_ELBO\nFrom the above I did not follow (and did not understand well) the impact of the first term in L_A in eq. (7)\n\n- Another issue that is not clear is written in lines 210-215. By the way, a difficult paragraph to follow.\nIn the mentioned paragraph it is said that:\n- \\hat{I^j}_m is generate from  \\hat{z^j}_c (line 210)\n- \\hat{I^j}_m is generated from \\hat{I^j}_o (line 211)\nSo, \\hat{I^j}_m has two different sources of generation ? I miss something here \n-It would be nice to introduce a “break point” from which the algorithm stars to fail, and to relate this to the GMM …\n\nMinor comments:\nI think the acronym “PGM” (line 69) is not introduced previously.\n", "limitations": "See my questions above.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\nThis paper contributes to the pluralistic image completion by using a grounded theoretical framework. This is achieved in an elegant way by using a GMM approach as a way to meet diversity. Also, this framework seems to be a generalization of [52], where the latter is focused on only one mixture component of the Normal distribution (unimodal Gaussian) with predefined set parameters  \nWeaknesses\nAlthough I think this is a valuable contribution to this research field, I have some questions that perhaps constitute the main weaknesses of the present paper.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "CNvx_A8y1Cg", "reviewer": "Reviewer_oAsX", "summary": "Problem: The paper deals with the problem of pluralistic image completion. This means that we want to generate multiple possible complete images.\n\nMethodology: The authors propose a method which uses probabilistic graph models for the problem. This allows to handle visual reality and diversity explicitly by adding constraints and using a Gaussian mixture model (GMM).\n \nExperiments: The proposed method produces good(most often state-of-the-art) results on various datasets like CelebA-hQ, Places2 and ImageNet both in qualitative and quantitative terms.\n\n########## POST REBUTTAL ###\nI have updated my rating as the authors have addressed my 2 key concerns.", "questions": "1. Please don't use the phrase 'Mixture of experts' because mixture of experts means something else in deep learning these days. It refers to models like LiMoE and Switch Transformer[B]. I don't see why you cannot just use GMM because that is all you are using. Can the authors please give a justification as to why this term is used?\n2. Ablation studies: Can the authors provide some ablation studies that will help use understand the contribution of the different components? For instance, the effect of the different losses used.\n\n\n[A] LIMoE: Learning Multiple Modalities with One Sparse Mixture-of-Experts Model\n\n[B] Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity\nWilliam Fedus, Barret Zoph, Noam Shazeer\nhttps://arxiv.org/abs/2101.03961 \n\n", "limitations": "No, not at all. I am quite surprised that a GAN paper is not discussing negative impacts despite all the deep fakes that have been propagation around and bringing bad publicity to AI.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Problem Relevance: The problem is quite relevant because diverse generation is more important than unique generation.\n2. Quality: I like the way the method section has been structured. Starting from the Objective ( sec 3.2) is a great idea. Because this allows future work to also build up on the objective and come up with different ways to solve it.\n3. Clarity: The paper is well written and clear.\n\nWeaknesses:\n1. Missing ablation studies\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "6S3WvE8DVz", "reviewer": "Reviewer_cWnv", "summary": "This paper proposes an end-to-end probabilistic method. The authors introduce a unified probabilistic graph model that represents the complex interactions in image completion. The entire procedure of image completion is then mathematically divided into several sub-procedures, which helps efficient enforcement of constraints. The sub-procedure directly related to pluralistic results is identified, where the interaction is established by a Gaussian mixture model (GMM). The inherent parameters of GMM are task-related, which are optimized adaptively during training, while the number of its primitives can control the diversity of results conveniently. The proposed method reaches state-of-the-art performance on various datasets.", "questions": "See above.", "limitations": "See above.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The theoretical proof is well conducted.\n- Experiments are convincing\n\n\nWeakness:\n- The shown figures are mostly low-resolution. Can the proposed method generate high-resolution images?\n- Lack of comparison with other generative models. E.g., GAN/VAE-based or diffusion models.\n- What is the advantages of using Mixture-of-Experts to do variational inference compared to traditional variational models (e.g., VAE)?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept