Image Inpainting via Iteratively Decoupled Probabilistic Modeling
Authors: Wenbo Li
Xin Yu
Kun Zhou
Yibing Song
Zhe Lin
Venue: ICLR 2024 spotlight (2024)
Keywords: Inpainting
Decoupled Probabilistic Modeling
Pixel Spread Model
Primary area: generative models

Abstract: Generative adversarial networks (GANs) have made great success in image inpainting yet still have difficulties tackling large missing regions. In contrast, iterative probabilistic algorithms, such as autoregressive and denoising diffusion models, have to be deployed with massive computing resources for decent effect. To achieve high-quality results with low computational cost, we present a novel pixel spread model (PSM) that iteratively employs decoupled probabilistic modeling, combining the optimization efficiency of GANs with the prediction tractability of probabilistic models. As a result, our model selectively spreads informative pixels throughout the image in a few iterations, largely enhancing the completion quality and efficiency. On multiple benchmarks, we achieve new state-of-the-art performance. Our code and models will be publicly available.

Published as a conference paper at ICLR 2024
IMAGE INPAINTING VIA ITERATIVELY DECOUPLED PROBABILISTIC MODELING
Wenbo Li', Xin Yu², Kun Zhou Yibing Song', Zhe Lin Huawei Noah's Ark Lab, 2HKU, CCUHK (SZ), AAlihaha DAMO Academy, 5 Adobe Research (fenglingiwb, yuxin27g, 2008kun303808, yibinggons e com
ABSTRACT
Generative adversarial networks (GANs) have made great success in image in- painting yet still have difficulties tackling large missing regions. In contrast, itera- tive probabilistic algorithms, such as autoregressive and denoising diffusion mod- els, have be deployed with massive computing resources for decent effect. To achieve high-quality results with low computational cost, we present novel pixel spread model (PSM) that iteratively employs decoupled probabilistic modeling, combining the optimization efficiency of GANs with the prediction tractability probabilistic models. As a result, our model selectively spreads informative pixels throughout the image in a few iterations, largely enhancing the completion quality and efficiency. On multiple benchmarks, we achieve new state-of-the-arr performance. Our code and models will be publicly available.
INTRODUCTION
Image inpainting, a fundamental computer vision task, aims to fill the missing regions in an image with visually pleasing and semantically appropriate content. It has been extensively employed in graphics and imaging applications, such as photo restoration Wan et al. (2020) 2022), image edit- ing Barnes al. 2009); Jo & Park (2019). compositing Levin al. (2004), re- re-targeting Cho et al. (2017). and object removal Criminisi et al. (2004). This task, especially filling large holes, is more ill-posed than other restoration problems, necessitating models of stronger generation abilities.
In past years, generative adversarial networks (GANs) have made great processes in image inpaint- ing Pathak et al. ()016); Yan et al. (2018); | Yu et al. (2018): [iu et al. (2019); Wan et al. (202T) ) et al. Chu et al. (2023); Sargsyan al. (2023). ). By implicitly modeling a target distribution through a main-max game, GANs-based methods significantly outperform traditional exemplar-base
techniques Hays & Efros (2007); Sun et al. (2005); Criminisi et al. (2004) 2003) in terms of visual quality. However, the one-shot generation of GANs sometimes lead to unstable training [Salimans) et al. (2016); Gulrajani al. (2017); Kodali et al. (2017) and makes it challenging to learn complex distribution, particularly when inpainting large holes in high-resolution images.
Conversely, autoregressive models Van den Oord et al. (2016); Van Den Oord et al. (2016); 6 Parmar let al. (2018) and denoising diffusion models [Song & Ermon (009); Ho et al. (2020); Dhariwal & Nichol| ) recently demonstrated remarkable power in content generation Ramesh et al. (2022);
Saharia et al. (202bb); Yu et al. (2022); Singer et al. (2022). These models utilize tractable proba- bilistic modeling techniques iteratively refine the image based on prior estimations, resulting in more stable training and improved coverage. However, it is widely known that autoregressive mod- els process images pixel by pixel. which makes it cumbersome handle high-resolution data, On the other hand, denoising diffusion models typically require thousands of iterations to achieve accu- rate estimations. Thus, using these methods directly image inpainting incurs respective drawbacks strategies for
large-hole high-resolution image inpainting still fall short.
To complete the map of inpainting, in this paper, we develop a new pixel spread model (PSM) tai- lored for the large-hole scenario. PSM operates in iterative manner, where all pixels are predicted in parallel during each iteration, and only qualified predictions are retained for subsequent iterations. It acts process gradually spread trustful pixels to unknown locations. Our core design lies in a simple yet highly effective decoupled probabilistic modeling (see Section [.I.II. which enjoys the merits of GANs' efficient optimization and the tractability of probabilistic models. In detail, our

---

Published as a conference paper at ICLR 2024
Our model supports photo-realistic large-hole inpainting for various scenarios. The first example for object removal is a high-resolution image captured in the wild, while others (512 x 512) come from Places2 Zhou al. (2017) and CelebA-HQ Karras al. (2018) datasets.
Figure 1:
model simultaneously predicts an inpainted result (i.e., the mean term) and uncertainty map (i.e., the variance term). The mean term is optimized using implicit adversarial training, yielding more accurate predictions with fewer iterations. The variance term, contrarily, is modeled explicitly using Gaussian regularization.
The adoption of our decoupled strategy offers numerous advantages. First, the use of adversarial optimization leads to significant reduction in the number of iterative steps required to achieve promising results, as shown Figure much faster than autoregressive and denoising diffusion models. Second, the Gaussian regulanization employed produces variance term that naturally acts as an uncertainty measure (see Section 3.1.2.. This allows for the selection of reliable estimates for iterative refinement, largely reducing GANs' artifacts. Furthermore, the explicit modeling of the distribution facilitates continuous sampling, thereby producing predictions with enhanced quality and diversity, as demonstrated in Section Ultimately, the uncertainty measure is instrumental in constructing an uncertainty-guider attention mechanism (see Section 3.2). which encourages the network to leverage more informative pixels for efficient reasoning. As result, our PSM completes large missing regions with photo-realistic content, as illustrated in Figure
Our contributions can be summarized as follows:
We develop novel pixel spread model (PSM) customized for large-hole hole- image inpainting. Thanks to the proposed iteratively decoupled probabilistic modeling, our model achieves efficient optimization and high-quality completion.
Our method reaches cutting-edge performance on both Places Zhou et al. (2017) and CelebA-HQ et al. (2018) benchmark datasets. Notably, our PSM outperforms popu- lar denoising diffusion models, e.g., LDM Rombach et al. (2022), by large margin, yield- ing 1.1 FID improvement on Places2 /Zhou al. (2017) while being significantly more weighte (only 20% parameters, 10x faster).
2 RELATED WORK
2.1 TRADITIONAL METHODS
Image inpainting is a classical computer vision problem. Early methods make use of image priors, such self-similarity and sparsity. Diffusion-baset methods Bertalmio et al. (2000): Ballester et al. 20011, for instance, convey information to the holes from nearby undamaged neighbors. Another line of exemplar based approaches Hays & Efros (007);; Sun et al. (005); Le Meur et al. (2011) Criminisi et al. (2003); Ding et al. (2018); Lee et al. (2016) looks for highly similar patches to complete missing regions using human-defined distance metrics. The most representative work is
2

---

Published as a conference paper at ICLR 2024
PatchMatch Barnes al. (2009), which employs heuristic searching in multi-scale image space to speed up inpainting greatly. However, due to lack of context understanding, they do not guarantee visually appealing and semantically consistent results.
2.2 DEEP LEARNING BASED METHODS
Using a great amount of training data to considerably increase the ability of high-level understand- ing, deee--eerrll-eewwee network-based methods Pathak al. (016); Yan et al. (2018); Zeng et al. (2019); Liu et al. (2020); Wang et al. (2018b) achieve success. Pathak et al. Pathak et al. (2016) introduce the adversarial loss Goodfellow et al. (014)) to inpainting, yielding visually realistic results. Several approaches along this line continually push the performance to new heights. For example, in order to obtain locally fine-grained details and globally consistent structures, lizuka et al. Tizuka et al. (2177) adopt two discriminators for adversarial training. Additionally, partial Liu et al. (2018) and gated Yu al. (2019) convolution layers are proposed to reduce artifacts, e.g., color discrepancy and blurriness, for irregular masks. Moreover, intermediate cues, including foreground contours Xiong iet al. (2019). object structures Nazeri et al.] (2019); Ren et al. (2019), and segmentation maps SSog al. (2088) are used in multi-stage generation. Despite nice inpainting content for small masks, these methods still do not guarantee large-hole inpainting quality.
2.3 LARGE HOLE IMAGE INPAINTING
To deal with large missing regions, a surge of effort was made improve the model capability.
Attention techniques Yu et al. (2018) Liu et al. 20 Xie et al.] (2019); Yi et al. (2020) and transformer architectures Wan al. (202 1: Zheng et al. (2021) : Li et al. (222); Ko & Kiml (2023)
take advantage of context information. They work well when an image contains repeating patterns. Besides, Zhao al. /Zhao et al. (2020) propose a novel architecture, bridging the gap between image- conditional and unconditional generation, improving free-form large-scale image completion. There are also attempts to study the progressive generation. This line is to select only high-quality pixels each time and gradually fill holes. We note that these methods heavily rely on specially designed update algorithms Zhang et al. (2018a): Guo et al. (2019); Li et al. (2020); [Oh et al. (2019), or consume additional model capacity separately assess the prediction accuracy Zeng et al. (2020).
or need more training stages Chang et al. 2022 when processing images.
Recently, benefiting from exact likelihood computation and iterative samplings, autoregressive mod- els Wan et al. (202 1 Yu et al. (202 Wu et al. (2022) and denoising diffusion models Saharia et al. (022a);; Rombach al. (2022) ; Lugmayr al. (2022) Avrahami al. (2022); Zhang et al. (2023) have shown great potential in producing diversified and realistic content. They inevitably incur high inference costs with thousands of steps and require massive computation resources. In this work, we present decoupled probabilistic modeling that obtains predictions and uncertainty measures simul- taneously. Our model identifies reliable predicted pixels and sends them to subsequent iterations, thereby mitigating GANs-generated artifacts. Also, the proposed approach can be viewed as a dif- fusion model that Iearns pixel spreading rather than denoising and requires fewer iterations.
OUR METHOD
Our objective to use photo-realistic material complete a masked image with substantial missing areas. In this section, we first formulate our pixel spread model (PSM) along with comprehensive analysis. followed by the details of model design and loss functions.
3.1 PIXEL SPREAD MODEL.
Although GANs-based methods achieve significantly better results than traditional ones, they still face great difficulties handling large missing regions. We attribute one of the reasons to the one-shot nature of GANs and instead propose iterative inpainting.
In each pass, since there are inevitably some good predictions, we use these pixels as clues to assist the next-time generation. In this way, our pixel spread model gradually propagates valuable information to the entire image. In the following, we first discuss the single-pass modeling before moving the pixel spread process.
3

---

Published as a conference paper at ICLR 2024
(Sec.3.1 3.1..3.1.2)
(Sec.3.1.1)
Plek & Sample (Uneertaint--Gaided)
Decoupled
Probabilistic
Modeling
Xy
(Sec.3.2)
Implicit
Modeling
X
Explicit
o
Regularization
Shared Weights
J o)
Xg
f1
Figure 2: Our pixel spread model for high-quality large-hole image inpainting. Left illustration is the pixel spread pipeline with proposed decoupled probabilistic modeling, and the right images are visual examples. We simplify the input of the t-th iteration 21-1. and denote the estimated mean and variance as μ4 and of- The map on the right is normalized for better visualization. We observe gradual uncertainty reduction in missing regions during the pixel spread process.
3.1.1 DECOUPLED PROBABILISTIC MODELING
For iterative inpainting, it is essential to find a mechanism to evaluate the accuracy of predictions. One intuitive solution introducing tractable probabilistic model so that uncertainty informa- tion can be analytically calculated. However, this requirement often leads to the assumption that the approximated target distribution Gaussian, which considerably too simple explain the truly complicated distributions. Although iterative models like denoising diffusion models Ho et al.) (2020) enhance marginal distribution expression by including a number hidden variables and op- timizing the variational lower evidence bound, these methods typically yield high inference cost.
To address these key issues, we propose decoupled probabilistic modeling tailored for efficient iterative inpainting. The essential insight is that we leverage the advantages of implicit GANs-basee optimization and explicit Gaussian regularization in a decoupled way. Thus we can simultancously obtain accurate predictions and explicit uncertainty measures.
As shown in Figure 2, given an input image I;-1 at time with large holes, our model (see archi- tecture details in Section 3.2) predicts the inpainting result μ4 well uncertainty map of. We use the adversarial loss (along with other losses Section (.3) to supervise image prediction fat, while jointly treating (1,, af) as the mean and diagonal covariance of Gaussian distribution. GANs' implicit optimization makes it possible to approximate the true distribution as closely as possible, greatly reducing the number of iterations. It also supplies with explicit uncertainty measure for the mean term, allowing us to select reliable pixels. The Gaussian regularization is mainly applied the variance term using negative log likelihood (NLL) Luul as
D
( N (x: ( dz,
C.u Σ
log
(I)
(wr)
where D is the data dimension and is the pixel index, 0 denotes model parameters, input and ground truth y are scaled to [-1, 1]. and follows the obtained Gaussian distribution. N. 8, (v) and 8_ (y) are defined
84( = {
8 if y
(2)
y + 255
if
1,
8_ (v)
00
if y
-1,
(3)
255
ify
>
Specifically, we include a stop-gradient operation (i.e., sgl:)), which encourages the Gaussian straint only to optimize the variance term and enables the mean term to be more accurately estimated through implicit modeling.
Discussion. We use the estimated mean and variance for sampling during the diffusion process, while taking the deterministic mean term the output for the final iteration. The feasibility of
4

---

Published as a conference paper at ICLR 2024
this design is proved by the experiments in Section 4 Additionally, the probabilistic modeling enables us to apply continuous sampling during pixel spread, yielding higher quality and more diverse estimations. Finally, we find the uncertainty measure also enables us to design more effective attention mechanism in Section 3.2
3.1.2 PIXEL SPREAD SCHEME
We use a feed-forward network, denoted as fo(-). gradually spread informative pixels the entire image, starting from known regions as
(4) where / is the time step, a, refers to the masked image, m. stands for a binary mask (1 for valid pixels while 0 for missing regions), and u, is the uncertainty map. The output includes the updated image, mask, and uncertainty map. Network parameters are shared across all iterations.
Lt. M,; U₁ = ( M,-1, u--1)
We use several iterations for both training and testing to improve performance. Specifically, as shown in Figure 2 and Eq. our method runs as follows the f-th iteration.
1. Predict. Given the masked image 2;-1, mask m,-1, and uncertainty map u,-1, our method estimates mean μ4 and variance o² for all pixels. Then preliminary uncertainty map ü, scaled to [0, 1] is generated by subtracting o, 's min value and dividing by absolute max-min value. Note that the values of a, remain unchanged.
2. Pick. We first sort the uncertainty scores for missing regions based on m According to the pre-defined mask schedule, we calculate the number of pixels that will be added in this iteration, and insert those with the lowest uncertainty to the known category, updating the mask to my Based on the preliminary uncertainty map ü, by marking locations that are still missing as I and the initially known pixels as O. while keeping ú, values of inpainted pixels up to this iteration (referring to m. mo). we obtain the final uncertainty map ue.
3. Sample. We consider
situations. First, for the initially known locations based on mp. we always use the original input pixels 2o (0 for missing regions while other pixels are valid). Second, we apply continuous sampling in accordance with fe, and or for the inpainting areas (referring to m. mo). The result is formulated as
(5) where a is adjustable ratio and z (O, I), and @ denotes Hadamard product. Note that the previously inpainted content gets updated at each iteration to maintain consistency with the newly inpainted pixels, and we do not use the o,z term in the final iteration.
= Fg + (m, - mo) @ (Ie + a ax.a; S z)
3.2 MODEL ARCHITECTURE
We use a deep U-Net Ronneberger et al. 2015) architecture with a StyleGAN Karras et al. (2019) 2020b) decoder, reaching large receptive fields with stacked convolutions to leverage context formation in images Buades et al. (2005); Mairal et al.] (2009): Berman et al. (2016) Wang et al. (018a)) In addition, we adopt multiple attention blocks at various resolutions, in light of the dis- covery that global interaction significantly improves reconstruction quality on much larger and more diverse datasets higher resolutions Yu et al. (2018) 8) Yi et al. (2020): Dhariwal Nichol (00T))
Based only on feature similarity, the conventional attention mechanism Vaswani et al. (2017) offers equal opportunity for pixels to exchange information. For the inpainting task, however, missing pixels are initialized with the same specified values, making them close to one another. As result, it is usually unable to effectively leverage useful information from visible regions. Even worse, the valid pixels are compromised, resulting in blurry content and unpleasing artifacts.
In this situation, as shown in Figure 3 we take into account the pixels' uncertainty scores to adjust the aggregating weights in attention by introducing a learnable function F. It properly resolves the problem mentioned above. The attention output is computed by
gk" VdL +
Attention(g. k u) Softmax
(6)
where (q. k. w are query, key, value matrices, dy denotes the scaling factor, and F predicts biased pixel weights using uncertainty map u and includes reshape operation.
5

---

Published as a conference paper at ICLR 2024
1--1
Re
Uncertainty-
Convolution
Guided Attention
Global Pooling
Skip Connection
de
Mappings
Figure 3: U-Net architecture with uncertainty- attention. We omit the mask update for clarity.
The o, map is normalized for better visualization.
3.3 LOss FUNCTIONS
In each iteration, our model outputs the mean and variance estimates, as shown in Figure 2 The mean term is optimized using adversarial loss Goodfflooe al. (2014) Lade and perceptual loss Su- vorov et al. (2021); 1 Johnson et al. (2016) Lpcp, which aims to produce natural-looking images. The losses are described as follows.
Adversarial loss. We formulate the adversarial loss as
Cug = EE [log [D (&))
(7)
Lad E.g llog [D (x)) Es [log (1 D (f))]
where is the discriminator Karras et al. (0119) I and à are real and predicted images.
Perceptual loss. We adopt high receptive field perceptual loss Suvorov al. (202
Lpep Σ (((()))
(x)))
where db is the layer output a pre-trained ResNet50 He et al. (0016).
As discussed in Section 3.1.1 we apply the negative log likelihood Lunt constrain the variance for uncertainty modeling. Thus the final loss function for the generator
L C= =,E,
Ai d2 + XaL nil
(10)
where j is the number of spread iterations. We empirically set dr 1, X2 = and Ag to 10-4.
EXPERIMENTS
4.1 DATASETS AND METRICS
We train our models at 512 x 512 resolution on Places2 (Zhou et al. (2017) and CelebA-HQ Karras al. (2018) in order to adequately assess the proposed method. Places2 large-scale dataset with nearly 8 million training images in various scene categories. Additionally, 36,500 images make up the validation split. During training, images undergo random flipping, cropping, and padding, while testing images are centrally cropped to the 512 x 512 size. For CelebA- HQQ. we employ 24,183
and 2,993 images, respectively. to train and our models. Following Yu al. (2019) Zhao al. (220): Suvorov et al. (202T): Li et al. (2022). we use on-the-fly generated masks during training, where the detailed setup from MAT et (2022). We evaluate all models using identical masks provided by Li et al. (2022) for fair comparisons. Besides, for evaluating model robustness, the same model to inpaint both small and large masks.
Despite being adopted in early inpainting work, LI distance, PSNR, and SSIM Wang al. (2004) are found not strongly associated with human perception when assessing image quality Ledig et al. 2017); Sajjadi et al. (2017). In this work, in light of Zhao al. (2020); Li al. (2022), HeDiHeusel et al. (2017). PDDDS Zhao et al. (2020), and U-IDS et al. (2018b). which robustly measures the perceptual fidelity of inpainted images, more suitable metrics.

---

Published as a conference paper at ICLR 2024
Table 1: Quantitative ablation study. Model "A" is the full model. Models "B" and "C" use fewer training iterations. We remove the decoupled probabilistic modeling (DPM), continuous sam- pling (CS), and uncertainty-guided attention (UGA) in models "D", "E", and "F". Model "G" adopts attention at 16 x 16 size.
Table 2: Quantitative ablation
study of the number of testing iterations. As the number of iterations increases, the FID
gets better and then saturates.
Model
Train lter.
DPM
CS /
UGA
Att. Res.
FID.
Test
Model A
Model
A
3
/
32,16
2.36
4
2.23
2.27
GH
1
/
32,16
2.95
5
2.16
2.20
2
32,16
2.55
6
2.12
2.16
D
3
32,16
2.49
2.09
2.14
E
3
/
32,16
2.45
2.07
2.12
F
3
/
32,16
2,44
9
2.05
2.11
G
3
V
/
16
2.64
10
2.05
2.11
4.2 IMPLEMENTATION DETAILS
We use an encoder-decoder architecture. The encoder made up of convolution blocks, while the decoder is adopted from StyleGAN2 Karras et al. (2020b). The encoder's channel size starts at 64 and doubles after each downsampling until the maximum of 512. The decoder has a symmetrical configuration. We adopt attention blocks at 32 32 and 16 16 resolutions. The uncertainty map initialized as "1 mask" at the first iteration. Given an H x W input, we first downsample the feature size to H. W before returning to H x W. More details are provided in Appendix A
32
We train our models for 20M images on Places2 and CelebA-HO using 8 NVIDIA A100 GPUs. We utilize exponential moving average (EMA), adaptive discriminator augmentation (ADA), and weight modulation training strategies Karras et al. (0200): Li et al. (2022). The batch size is 32, and the learning rate is 10-3 We employ an Adam Kingma & Ba 2015) optimizer with
0 and 32 = 0.99. We empirically set a 0.01 in Eq. ( based on experimental results. During training, our model undergoes the entire pipeline with iterations to enhance efficiency. However, during testing, the model iterates four times to achieve improved restoration results.
The fact that previous work Zhao et al. (2020) Li et al. (2022) trains models on Places2 with 50M or more images- much more extensive data than ours evidences the benefit of our method. Addi- tional training can further improve our approach, and yet 20M images already deliver cutting-edge performance. Our model's generalization ability is demonstrated in Appendices Baand
4.3 ABLATION STUDY
For quick evaluation, we train our models for 6M images at 256 x 256 resolution using Places365- Standard, subset of Places2 Zhou et al. (2017). We start with model "A", which employs our full designs and adopts three iterations during training.
Iterative number. Our core idea is employ iterative optimization enhance the generation quality. We adjust the iteration number and maintain the same setup during training and testing. As illustrated in Table models with and two iterations, dubbed "B" and "C", yield 0.59 and 0.19 FID decreases compared to model "A". Also, as shown in Figure K. adopting more iterations is capable producing more aesthetically pleasing content. The first and third cases exhibit obviously fewer artifacts, and the arch in the second example is successfully restored after three iterations.
is noted that we can test the system with a different iteration number from the training stage. Using more iterations results in higher FID performance, as demonstrated in Table 2 yet at the expense of longer inference time. Thus, there is a trade-off between inference speed and generation quality. Additionally, when comparing models "A" and "B", it is clear that introducing more iterations in the training process is beneficial. But the number of iterations in the inference stage more important.
Decoupled probabilistic modeling. To deliver accurate prediction while supporting the uncertainty measure for iterative inpainting, we propose decoupled probabilistic modeling. When putting all supervision on the sampled result, we observe the training diminishes the variance term (close to 0 for all pixels). It is because, unlike denoising diffusion models that precisely quantify the noise levels at each step, GANs-based method no longer provides specific optimization targets for the mean and variance terms. The variance term is underestimated for trivial optimization in this case. It renders the picking process less effective.
7

---

Published as a conference paper at ICLR 2024
Input
Model D DPM)
Model A (Full)
GT
0.8
Mask Ratio
0.6
0.4
Input
Model (
Model (Full)
cosine
linear
root
0.0
0.0
0.2
0.4
0.6
0.8
1.0
Figure 5: Visualization of mask schedule functions.
Input
Model F (- UGA)
Model (Full)
Table 3: Ablation study of mask schedule functions.
Mask Sche.
Iter.
FID
Cubic
2.54
Figure Qualitative ablation study. Model "A" is the full model. The proposed decoupled probabilistic modeling, contin- uous sampling, and uncertainty- guuiddd guided attention designs are not used in models "D", "E", and "F"
Cosine
2.48
Linear
2.36
Square Root
2.47
As illustrated in Table [ model "D" obtains an inferior FID result compared with the full model "A". Besides, from the visual comparison in Figure it is observed that model "D" tends generate blurry content, while model "A" produces sharper structures and fine-grained details.
Continuous sampling. Our approach uses the estimated variance perform continuous sampling- Table [ indicates that FID decreases by nearly 0.1 when continuous sampling (model "E") is not involved. Also, it is observed that our full model leads more visually consistent content. For example, box structures are well restored from the visible pixels in Figure 4 Thus, continuous sampling brings higher fidelity to our results. As shown in Figure L.2] our model also supports the pluralistic generation, particularly in the hole's center. However, when the mean term is estimated with low uncertainty or the iteration number constrained, the differences are not always instantly obvious. A detailed analysis of -idelity-diversity trade-off is further provided in Appendix H
Uncertainty-guided attention. To fully exploit distant context, we add attention blocks our framework. We first compare using attention 32 x 32, 16 x 16 (model "A") and only at 16 x 16 (model "G"). We discover a 0.28 FID drop in model "G" from the quantitative comparison in Table demonstrating the significance of long-range interaction in large-hole image inpainting.
Besides, as aforementioned Section the conventional attention mechanism may result in color consistency and blurriness. To support this claim, tease apart the uncertainty guidance and notice a minor performance drop in Table Also, we provide a visual comparison in Figure 4 We observe that model "A" produces more visually appealing window details than model "F".
Mask schedule. As illustrated Table and Figure 5 we analyze various mask schedule strategies and discover that the uniform strategy performs best. We argue this because the mask ratios of in- put images vary widely, and uniform schedule results in more stable training for different iterations.
4.4
COMPARISONS TO STATE-OF-THE-AN METHODS
We thoroughly compare the proposed pixel spread model (PSM) with GANs-based models Li et al.
2022 Zhao et al. 2020);; Suvorov et al. (2021): Zhu et al. (202T): Zeng et al. (2021): Yi et al.
2020 Yu et al. 2019), autoregressive models Wan al. (2021). l). and denoising diffusion mod- els Rombach et al. (2022) in Table 4 We use publicly accessible models for 512 x 512 resolution
and test them the masks make fair comparison.
8

---

Published as a conference paper at ICLR 2024
Table 4: Quantitative comparisons on Places2 and CelebA-HQ under 512 x 512 small and large mask settings. "f": Stable Diffusion inpainting model trained on LAION-Aestheti V2 5+. P-IDS and U-IDS are shown as percentages(%) The best and second best results are red and blue.
Places2 (612 612)
CelebA-HO (512
612)
#Param.
Method
Small Mask
Large Mask
Small Mask
Large Mask
x10
FID,
P-IDST
U-IDS
FIDJ
P-IDS1
U-IDS"
FID,
P-IDS1
U-IDS1
FID, P-IDST U-IDST
PSM (ours)
0.72
30.95
43.91
25.33
39.30
2.34
22.42
33.43
4.05
16.10
28.25
Stable Ditfusiion
860
1.32
12.69
34.78
2.11
12.01
32.57
LDM
387
1.06
16.23
39.61
2.76
12.11
33.02
MAT
62
1.07
27.42
41.93
2.90
19.03
35.36
2.86
21.15
32.56
4.86
13.83
25.33
CoModGAN
109
1.10
26.95
41.88
2.92
19.64
35.78
3.26
19.65
31.41
5.65
11.23
22.54
LaMa
51/27
0.99
22.79
40.58
2.97
13.09
32.29
4.05
9.72
21.57
8.15
2.07
7.58
MADF
85
2.24
14.85
35.03
7.53
6.00
23.78
3.39
12.06
24.61
6.83
3.41
11.26
AOT GAN
15
3.19
8.07
30,94
10.64
3.07
19.92
4.65
20.45
10.82
1.94
6.97
HFill
3
7.94
23.60
28.92
1.24
11.24
Input
Original
Big-LaMa
MAT
LDM
PSM (ours)
Figure 6: Qualitative comparisons of state-of of-the- methods on 512 x 512 Places2. Our PSM produces structures and details that are more realistic and reasonable. Best viewed zoomed in.
In Table 4 our method significantly performs better than the existing GANs-based models under both large and small mask settings. Besides, even with only 20% of the parameters of strong de- noising diffusion model LDM Rombach et al. (202)). our method delivers superior results in terms of all metrics. For example, on the Places2 benchmark, our PSM brings about 1.1 improvement on FID and larger gains on P-IDS and U-ID under the large mask setup- As for the inference speed, our PSM costs nearly 250ms obtain Notably, model is trained using far fewer samples (our 20M images vs. CoModGAN's Zhao ctt al. 2020) 50M images). Further comparisons are presented in Appendices D
x 512 image, which 10x faster than LDM (~~3s).
We also provide visual comparisons in Figure Appendix N In variety of scenes, our method generates more aesthetically pleasing textures with fewer artifacts when compared to existing meth- ods. For instance, room layouts and building structures are better inpainted by our approach.
CONCLUSION
We have proposed new pixel spread model for large-hole image inpainting. Utilizing the proposed iteratively decoupled probabilistic modeling, our method can assess the prediction accuracy and retain the pixels with the lowest uncertainty as hints for subsequent processing, yielding high-qualin completion. Furthermore, of prevalent denoising diffusion models. The state-of-the-a performance in multiple benchmarks demonstrates the effectiveness of our method. Lastly, we analyze limitations in Appendix M
method exhibits favorable inference efficiency, largely surpassing that

---

Published as a conference paper at ICLR 2024
REFERENCES
Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In CVPR. pp. 18208-18218, 2022.
Coloma Ballester, Marcelo Bertalmio, Vicent Caselles, Guillermo Sapiro, and Joan Verdera. Filling- in by joint interpolation of vector fields and gray levels. TIP, 10(8): 1200-121, 2001.
Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan Goldman. Patchmatch: A random- ized correspondence algorithm for structural image editing. TOG, 28())24, 2009.
Dana Berman, Shai Avidan, al. Non-local image dehazing. In CVPR, pp- 1674-1682, 2016.
Marcelo Bertalmio, Guillermo Sapiro, Vincent Caselles, and Coloma Ballester. Image inpainting. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pp. 417-424, 2000.
Antoni Buades, Bartomeu Coll, and J-M Morel. A non-local algorithm for image denoising. In CVPR, volume 2, pp. 60-65. IEEE, 2005.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In CVPR, pp. 11315-1
2022.
Donghyeon Cho, Jinsun Park, Tae-Hyun Oh, Yu- Yu-Wing Tai, and In So Kweon. Weakly-and self- supervised learning for content-aware deep image retargeting. In ICCV, pp. 4558-4567 2017.
Tianyi Chu, Jiafu Chen, Jiakai Sun, Shuobin Lian, Zhizhong Wang, Zhiwen Zuo, Lei Zhao, Wei Xing, and Dongming Lu. Rethinking fast fourier convolution image inpainting. In ICCV, pp. 23195-23205, 2023.
Antonio Criminisi, Patrick Perez, and Kentaro Toyama. Object removal by exemplar-based inpaint- ing. In CVPR, volume 2. pp. II-II. IEEE, 2003.
Antonio Criminisi, Patrick Pérez, and Kentaro Toyama. Region filling and object removal by exemplar-based image inpainting. 7IP, 1309):1200-1212, 2004.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. NIPS, 34: 8780-8794, 2021.
Ding Ding, Sundaresh Ram, and Jeffrey Rodríguez. Image inpainting using nonlocal texture matching and nonlinear filtering. TIP, 2805) 2018.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NIPS, 27, 2014.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Im- proved training of wasserstein gans. NIPS, 30, 2017.
Zongyu Guo, Zhibo Chen, Tao Yu, Jiale Chen, and Sen Liu. Progressive image inpainting with full-resolution residual network. In ACMMM, pp. 2496-2504 2019.
James Hays and Alexei A Efros. Scene completion using millions of photographs. ToG, 26(3):4-es, 2007.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pp. 770-778, 2016.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to local nash equilibrium. NIPS, 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NIPS, 33: 6840-6851, 2020.
Satoshi lizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. Globally and locally consistent image completion. ToG, 36(4):1-14, 2017.
10

---

Published as a conference paper at ICLR 2024
Youngjoo Jo and Jongyoul Park. Sc-fegan: Face editing generative adversarial network with user's sketch and color. In ICCV, pp. 1745-1753, 2019.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV, pp. 694-711. Springer. 2016.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. In ICLR, 2018.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, pp. 4401-4410 2019.
Tero Karras, Mikk Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. NIPS, 33:12104-12114, 2020a.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of stylegan. In CVPR, pp. 8110-81 10,8819, 2020b.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Keunsoo Ko and Chang-Su Kim. Continuously masked transformer for image inpainting. In ICCV, pp. 13169-13178, 2023.
Naveen Kodali, Jacob Abernethy, James Hays, and Zsolt Kira. On convergence and stability of gans. arXiv preprint arXiv: (005:v/700.072200 2017.
Olivier Le Meur, Josselin Gautier, and Christine Guillemot. Examplar-based inpainting based on local geometry. In ICIP, pp. 3401-3404. IEEE, 2011.
Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic single image super-resolution using generative adversarial network. In CVPR, pp. 4681 16990, 2017.
Joo Ho Lee, Inchang Choi, and Min H Kim. Laplacian patch- image synthesis. CVPR, pp. 2727-2735, 2016.
Anat Levin, Assaf Zomet, Shmuel Peleg, and Yair Weiss. Seamless image stitching in the gradient domain. In ECCV, pp. 377-389. Springer, 2004.
Jingyuan Li, Ning Wang. Lefei Zhang. Bo Du, and Dacheng Tao. Recurrent feature reasoning for image inpainting. In CVPR, pp. 7760-7768, 2020.
Wenbo Li, Zhe Lin, Kun Zhou, Lu Qi, Yi Wang, and Jiaya Jia. Mat: Mask-aware transformer for large hole image inpainting. In CVPR, pp. 10758-10768, 2022.
Guilin Liu, Fitsum Reda, Kevin Shih, Ting-Chun Wang, Andrew Tao, and Bryan Catanzaro. Image inpainting for irregular holes using partial convolutions. In ECCV, pp. 85-100, 2018.
Hongyu Liu, Bin Jiang, Yi Xiao, and Chao Yang. Coherent semantic attention for image inpainting. In ICCV, pp. 4170-4179, 2019.
Hongyu Liu, Bin Jiang, Yibing Song, Wei Huang, and Chao Yang. Rethinking image inpainting via mutual encoder-decoder with feature equalizations. In ECCV, pp. 725-741. Springer, 2020.
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In CVPR, pp. 11461-11471, 2022.
Julien Mairal, Francis Bach, Jean Ponce, Guillermo Sapiro, and Andrew Zisserman. Non-local sparse models for image restoration. In ICCV, pp. 2272-2279, IEEE, 2009.
Kamyar Nazeri, Eric Ng, Tony Joseph, Faisal Z Qureshi, and Mehran Ebrahimi. Edgeconnect: Generative image inpainting with adversarial edge learning. arXiv preprint arXiv: 0rXiv:1911.00212 2019.
11

---

Published as a conference paper at ICLR 2024
Scoung Wug Oh, Sungho Lee, Joon- Joon-Young Lee, and Seon Joo Kim. Onion-peel networks for deep video completion. In ICCV, pp. 4403 4412, 2019.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In ICML, pp. 4053-4064, PMLR, 2018.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In CVPR, pp. 2536-2544, 2016.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.0611 2022.
Yurui Ren, Xiaoming Yu, Ruonan Zhang, Thomas H Li, Shan Liu, and Ge Li. Structureflow: Image inpainting via structure-aware appearance flow. In ICCV, pp. 181-190, 2019.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, pp. 10684-10695, 2022.
Andres Romero, Angela Castillo, Jose Abril-Nova, Radu Timofte, Ritwik Das, Sanchit Hira, Zhi- hong Pan, Min Zhang, Baopu Li, Dongliang Hc, et al. Ntire 2022 image inpainting challenge: Report. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recogni- tion, pp. 1150-1180 1122, 2022.
Olaf Ronneherger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234-241. Springer, 2015.
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans. David Fleet, and Mohammad Norouzi. Palette: image-to-image diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings, pp. 1-10, 2022a.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205. 1487, 2022b.
Mehdi SM Sajjadi, Bernhard Scholkopf, and Michael Hirsch. Enhancenet: Single image super- resolution through automated texture synthesis. In ICCV, pp. 4491-4500, 2017.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. NIPS, 29, 2016.
Andraník Sargsyan, Shant Navasardyan, Xingqian Xu, and Humphrey Shi. Mi-gan: A simple base- line for image inpainting on mobile devices. In ICCV, pp. 7335-7345, 2023.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint 224V::2299.14999 2022.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. NIPS. 32, 2019.
Yuhang Song, Chao Yang. Yeji Shen, Peng Wang. Qin Huang, and C-C Jay Kuo. Spg-net: Segmen- tation prediction and guidance network for image inpainting. arXiv preprint arXiv:1805.03356, 2018.
Jian Sun, Lu Yuan, Jiaya Jia, and fleung-Yeurn Shum. Image completion with structure propagation. In ACM SIGGRAPH 2005 Papers, pp. 861-868. 2005.
Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka. Kiwoong Park, and Victor Lempit- sky. Resolution-robust large mask inpainting with fourier convolutions. arXiv preprint 10X.v22109.07161, 2021.
12

---

Published as a conference paper at ICLR 2024
Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Condi- tional image generation with pixelenn decoders. NIPS, 29, 2016.
Aäron Van Den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In ICML, pp. 1747-1756. PMLR, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention all you need. NIPS, pp. 5998-6008, 2017.
Ziyu Wan, Bo Zhang, Dongdong Chen, Pan Zhang, Dong Chen, Jing Liao, and Fang Wen. Bringing 2020.
old photos back to life. In CVPR. pp.
Ziyu Wan, Jingbo Zhang, Dongdong Chen, and Jing Liao. High-fidelity pluralistic image completion with transformers. arXiv preprint arXiv:2103. 14031, 2021.
Ziyu Wan, Bo Zhang, Dongdong Chen, Pan Zhang, Dong Chen, Fang Wen, and Jing Liao. Old photo restoration via deep latent space translation. TPAMI, 2022.
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR, pp. 7794-7803, 2018a.
Yi Wang, Xin Tao, Xiaojuan Qi, Xiaoyong Shen, and Jiaya Jia. Image inpainting via generative multi-column convolutional neural networks. NIPS, 2018b.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. T7P, 13(4):600-612 2004.
Chenfei Wu, Jian Liang, Xiaowei Hu, Zhe Gan, Jianfeng Wang. Lijuan Wang. Zicheng Liu, Yuejian Fang, and Nan Duan. Nuwa-infinity: Autoregressive over autoregressive generation for infinite visual synthesis. NIPS, 2022.
Chaohao Xie, Shaohui Liu, Chao Li, Ming-Ming Cheng, Wangmeng Zuo, Xiao Liu, Shilei Wen, and Errui Ding. Image inpainting with learnable bidirectional attention maps. In ICCV, pp. 8558-8867,2 2019.
Wei Xiong, Jiahui Yu, Zhe Lin, Jimei Yang. Xin Lu, Connelly Barnes, and Jiebo Luo. Foreground- aware image inpainting. In CVPR, pp. 5840-5848, 2019.
Zhaoyi Yan, Xiaoming Li, Mu Li, Wangmeng Zuo, and Shiguang Shan. Shift-net: Image inpainting via deep feature rearrangement. In ECCV, pp. 1-17, 2018.
Zili Yi, Qiang Tang, Shekoofeh Azizi, Dacsik Jang, and Zhan Xu. Contextual residual aggregation for ultra high-resolution image inpainting. In CVPR, pp. 7508-7517, 2020.
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Generative image inpainting with contextual attention. In CVPR, pp. 5505-5514, 2018.
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Free-form image inpainting with gated convolution. In ICCV, pp. 4471 14880 2019.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang. Bureu Karagol Ayan, al. Scaling autoregressive models for content- rich text-to-image generation. arXiv preprint arXiv:2206. 10789, 2022.
Yingchen Yu, Fangneng Zhan, Rongliang Wu, Jianxiong Pan, Kaiwen Cui, Shijian Lu, Feiying Ma, Xuansong Xie, and Chunyan Miao. Diverse image inpainting with bidirectional and autoregres- sive transformers. arXiv preprint arXiv:2104.1233 2021.
Yanhong Zeng, Jianlong Fu, Hongyang Chao, and Baining Guo. Learning pyramid-context encoder network for high-quality image inpainting. In CVPR, pp. 1486-1494, 2019.
Yanhong Zeng, Jianlong Fu, Hongyang Chao, and Baining Guo. Aggregated contextual transforma- tions for high-resolution image inpainting. arXiv preprint arXiv:2104.01431, 2021.
13

---

Published as a conference paper at ICLR 2024
Yu Zeng. Zhe Lin, Jimei Yang, Jianming Zhang, Eli Shechtman, and Huchuan Lu. High-resolution image inpainting with iterative confidence feedback and guided upsampling. In ECCV, pp. 1-17. Springer, 2020.
Haoran Zhang, Zhenzhen Hu, Changzhi Luo, Wangmeng Zuo, and Meng Wang. Semantic image inpainting with progressive generative networks. In ACMMM, pp. 1939-1947, 2018a.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. ICCV, pp. 3836-3847, 2023.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unrcasonable effectiveness of deep features perceptual metric. In CVPR, pp. 586-5956555 2018b.
Shengyu Zhao, Jonathan Cui, Yilun Sheng, Yue Dong, Xiao Liang, Eric, Chao Chang, and Yan Xu. Large scale image complction via co-modulated generative adversarial networks. In ICLR, 2020.
Chuanxia Zheng, Tat-Jen Cham, and Jianfei Cai. Tfill: Image completion via transformer-based architecture. arXiv preprint arXiv: 0040v22004.00845, 2021.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. PAMI, 40(6): 1452-1464, 2017.
Manyu Zhu, Dongliang He, Xin Li, Chao Li, Fu Li, Xiao Liu, Errui Ding, and Zhaoxiang Zhang. Image inpainting by end-to-end cascaded refinement with mask awareness. TIP, 30:4855-4866. 2021.
14

---

Published as a conference paper at ICLR 2024
A ARCHITECTURE DETAILS
we here provide more through il- lustration of architecture details. We adopt U-Net architecture with skip connections, where the encoder downsamples the size of an H W input to H to H x W. At each resolution, there is just one residual block made up of two 3 x 3 convolutional layers, unless otherwise stated. Both the encoder and the decoder employ attention blocks feature sizes of H W and H W and an early convolutional block is also introduced at these scales. I6 Different attention blocks use adaptive mapping functions in Figure 3 each of which is composed of 4 convolutional layers with kernel size of 3 3.
Apart from the descriptions in Section 3.2 and Section 4.2
32 32 and the decoder upsamples back
32 32.
The input consists of 7 channels: 3 for color images, for the initial mask, for the updated mask, 1 for the uncertainty map, and for the time step. The number of channels is initially converted to 64, then doubled after each downsampling, up to a maximum of 512, and the decoder employs a symmetrical setup. The output contains 6 channels: 3 for the mean term, for the log variance term.
We apply weight modulation, where the style derived from an image global feature and a random latent code. As for the global feature, we employ convolutional layers to further downsample the feature size from H W
W and global pooling layer to obtain Id representation. The random latent code is generated from Gaussian noise using 8 fully connected layers.
32 32 to 256 X 255
B GENERALIZATION TO 1024 1024 RESOLUTION
To evaluate the generalization ability of models, we compare our pixel spread model (PSM), MAT] Li al. (2022) and LaMa Suvorov al. (202T) trained on 512 x 512 Places2 Zhou et al. (2017) the 1024 x 1024 resolution. As illustrated in Table our PSM performs significantly better than MAT and LaMa on all metrics, despite using fewer training samples. Remarkably, our approach results in an approximately 1.9 FID improvement. We do not involve denoising diffusion models (e-g., LDM) and other GANs-based models (e.g., CoModGAN) for comparisons because scaling them up to the 1024 X 1024 resolution is impractical.
Table B. Quantitative comparisons on 1024 x 1024 Places2 Zhou et al. (217)) dataset under the large mask setup by transferring models trained at the 512 x 512 resolution. Our PSM generalizes well to higher resolutions.
Method
FID
P-IDS(%) U-IDS(%
PSM (Ours)
3.95
14.40
32.23
MAT et al. (2022
5,83
9.51
28.02
LaMa Suvorov et al. 22221
6.31
4.98
23.24
C GENERALIZATION TO UNKNOWN MASK TYPES
Following the NTIRE 2022 Image Inpainting Challenge Romero et al. (0222) guidelines, we prepare a test set of 6000 samples with an average missing ratio of approximately 60%, covering mask types of Every N Lines, Image Expansion, and Nearest Neighbor, which are never seen during training.
Koomero
Table C.2: Quantitative comparisons on unknown mask types.
Method
#Param.
FID|
P-IDSt
LPIPS
PSNR
Ours + Fine-tune
74M
2.22
26.43
0.091
26.30dB
Ours
74M
5.27
12.98
0.148
23.77dB
Stable Diffusion
860M
95.65
3.65
0.705
13.56dB
LDM
387M
96.39
5.48
0.576
13.
MAT
62M
67.57
3.28
0.458
16.25aB
Big LaMa
51M
47.64
3.97
0.306
20.41dB
15

---

Published as a conference paper at ICLR 2024
As shown in Table both other SOTA diffusion and GANs-based models show significant per- formance degradation, particularly for large models like Stable Diffusion and LDM. Surprisingly, method exhibits excellent generalization unknown mask types. Our FID score of 5.27 far surpasses the second-best method, LaMa, with a score of 47.64. We attribute this superiority to iteratively decoupled probabilistic modeling, which enables the selection of reliable estimates for iterative refinement. Moreover, fine-tuning with a few iterations leads to great gains. These findings manifest the exceptional generalization and optimization abilities of our method.
D 512x512 LPIPS RESULTS
LPIPS Zhang al. 2018b) is also widely used perceptual metric in image inpainting. For comprehensive comparison with state-of-the-art methods, we provide LPIPS results Table We argue that LPIPS may not be suitable for large-hole image inpainting because it is calculated pixel-by-pixel. This measure for reference only.
Table D.3: LPIPSJ results on 512 x 512 Places2 Zhou et al. (017)) and CelebA-HQ Karras et al. 2018) datasets. "t": models are trained with 20M samples, much less than other methods (e-g-- MAT uses 50M samples on Places2 and 25M samples on CelebA-HQ). We use single model for both the small and large mask setups. "g": the official Stable Diffusion inpainting model trained on a large-scale high-quality dataset LAION-Aesthetic, V2 5+.
#Pramm. 10
Places
CelebA-HO
Method
Small
Large
Small
Large
PSM (Ours)
74
0.084
0.161
0.052
0.099
Stable Diffusion
860
0.148
0.220
LDM Rombach et al. 2022
387
0.100
0.190
Li al. 2022
62
0.099
0.189
0.065
0.125
CoModGAN Zhao et al. 2020
109
0.101
0.192
0.073
0.140
LaMa Suvorov
2021
51/27
0.086
0.166
0.075
0.143
MADF Zhu al. 2021
85
0.095
0.181
0.068
0.130
AOT GAN Zeng et al. 2021
15
0.101
0.195
0.074
0.145
HFill Yi 2020
3
0.148
0.284
E 256x 256 CELEBA -HQ RESULTS
We also conduct quantitative comparisons on 256 256 CelebA-HQ Karras et al. (2018) dataset. As shown in Table E.4 our method achieves the best performance among all methods
Table E.4:
Quantitative comparisons on 256 256 CelebA-Ho Karras et al. (2018) dataset. The P-IDS and U-IDS results are shown in percentage (%). "f": our model is trained with 12M samples, far less than other methods (e.g., MAT uses 25M samples). We use a single model for both the small and large mask setups.
Small Mask
Large Mask
Method
FID
P-IDSt
U-IDS*
FID,
P-IDS
U-IDST
PSM (Ours)
2.58
21.35
33.70
4.57
14.07
25.28
MAT
Li et al.
2022
2.94
20.88
32.01
5.16
13.90
25.13
LaMa Suvorov et al. 2021
3.98
8.82
22.57
8.75
2.34
8.77
ICT Wan et al. 202T
5.24
4.51
17.39
10.92
0.90
5.23
RFR
Li et al. 2020
6.37
5.75
14.97
12.91
0.70
1.77
MADF Zhu ct al. 2021
10.43
6.25
14.62
23.59
0.50
1.44
AOT GAN Zeng
2022
9.64
5.61
14.62
22.91
0.47
1.65
DeepFill v2
Yu et al.
2019
5.69
6.62
16.82
13.23
0.84
2.62
EdgeConnect
Nazeri et al.
(001
5.24
5.61
15.65
12.16
0.84
2.31
16

---

Published as a conference paper at ICLR 2024
F 512x512 PSNR RESULTS AND INFERENCE SPEED
While PSNR may not be the most suitable metric for assessing large-scale hole inpainting per- formance, provide the results in Table F.5 for reference. It is worth noting that method demonstrates notably superior PSNR outcomes. In terms of inference efficiency, it is evident that model stands out for its efficiency among the top-performing models.
Table F.5: PSNR (dB) results and inference speed on 512 x 512 Places2 Zhou et al. (0117) and CelebA-HQ Karras et al. 2018) datasets.
Method
PSM (Ours)
Stable Diffusion
LDM
MAT
Small
25.51dB
21.70dB
24.48dB
24.44dB
Places
Large
20.89.IB
19.17dB
20.1
19.92dB
Small
29.61dB
28.44dB
CelebA-HQ
Large
24.81dB
23.50dB
Inference Speed
0.25s
3.6s
2.7s
0.26s
G COMPARISON TO REPAINT
Considering that the sizes of RePaint Lugmayr et al. (2022) results are at 256 x 256 on Places2 and CelebA-HQ while ours are 512 512, we don't compare it in the main body of the paper. Here we compare our model PSM to RePaint at 256 x 256 resolution on Places2 and CelebA-HQ in Table G.6 where PSM achieves better performance and is 1000 faster than RePaint (i.e., 0.25s v.8. 250s for one image processing). For saving time, we just use the first 10K Places2 validation images for evaluation.
Table G.6: Quantitative comparisons with RePaint Lugmayr et al. (0222 on 256 x 256 Places Zhou
al. (017) and CelebA-HQ Haaa--QQaaasss al. (2018) datasets.
Places2- 10K (256 x 256)
CelebA-HQ (256 x 256)
Method
FID P-IDS(%) U-IDS(%)T
FIDJ P-IDS(%) U-IDS(%)
Ours
3.47
18.32
34.52
4.57
14.07
25.28
RePaint
6.15
11.11
27.16
10.55
0.07
1.47
H FIDELITY-DIVERSITY TRADE-OFF
Apart from FID (depending on both diversity and fidelity), we follow previous work to use Improved Precision and Recall as fidelity (precision) and diversity (recall) measures. As shown in Table H.7 our model yields better FID, higher precision yet slightly lower recall than LDM on Places2, while outperforming MAT on all metrics. Improving diversity will be our future work.
Table H.7: FID, precision and recall comparisons for evaluating fidelity-diversity traed-off on 512 x 512 Places Zhou et al. (2017) dataset.
Method
#Param.
FID
Precision
Recall
PSM (Ours)
74M
1.68
0.983
0.971
LDM
387M
2.76
0.962
0.975
MAT
62M
2.90
0.965
0.939
17

---

Published as a conference paper at ICLR 2024
I ADDITIONAL COMPARISONS
We have expanded our comparisons to include recent methods, including MI-GAN Sargsyan et al. 2023) and the inpainting model from ControlNet Zhang al. (2023), depicted in Table L8 The results from our proposed PSM demonstrate significant improvements across all metrics, high- lighting its effectiveness. MI-GAN is primarily designed for mobile devices, achieving favorable performance-efficinne trade-off. Moreover, it is worth noting that ControlNet may produce subop- timal results due its tendency generate new objects that might not align harmoniously with the existing content.
Table 1.8: Quantitative comparisons on Places2 and CelebA-HQ under 512 512 small and large mask settings. "f" Stable Diffusion inpainting model trained LAION-Aesthetin V2 5+. P-IDS and U-IDS are shown as percentages(%)) ()). The best and second best results are red and blue.
Places2 (512 512)
CelebA-IIQ (512 x 512)
#Param.
Method
Small Mask
Large Mask
Small Mask
Large Mask
x10"
FID
P-IDSt
U-IDS"
FID/
P-IDST U-IDST
FIDI
P-IDST
U-IDST
FIDL P-IDST U-IDST
PSM (ours)
74
0.72
30.95
43.91
1.68
25.33
39.30
2.34
22.42
33.43
16.10
28.25
Stable Diflusion
860
1.32
12.69
34.78
2.11
12.01
32.57
LDM
387
1.06
16.23
39.61
2.76
12.11
33.02
1.07
27.42
41.93
19.03
35,36
2.86
21.15
32.56
13.83
25.33
CoModGAN
109
1.10
26.95
41.88
2.92
19.64
35.78
3.26
19.65
31.41
5.65
11.23
22.54
LaMa
51/27
0.99
22.79
40.58
2.97
13.09
32.29
4.05
9.72
21.57
8.15
2.07
7.58
MI-GAN
1.40
18.43
39.35
3.81
13.50
32.42
ControlNet
1223
1.86
12.63
35.71
5,55
6.60
25.65
MADF
85
2.24
14.85
35.03
7.53
6.00
23.78
3.39
12.06
24.61
6.83
3.41
11.26
AOT GAN
15
3.19
8.07
30.94
10.64
3.07
19.92
4.65
7.92
20.45
10.82
1.94
6.97
HRill
3
7.94
3.98
23.60
28.92
1.24
11.24
J RATIO a IN EQ.
From Table J.9 we that large generally trades fidelity (Precision) for higher diversity (Recall) on Places2. We empirically choose the a = 0.001 for better evaluation results.
Table J.9: Quantitative results using different values in Eq.
Value
0.001 (Ours)
0.1
FID | 1.68/0.9830.971
1.75/0.97700.975
K VISUAL ITERATION PROCESS
We depict the evolving results at various iterations in Figure K.1 It is evident that our method attains promising outcomes within limited number of iterations, significantly faster than autoregressive and denoising diffusion models. This point is already underscored in the speed comparison discussed in Section 4.4 and Appendix
L PLURALISTIC GENERATION
As discussed in Section our method also supports pluralistic generation. From the visual ex- amples Figure L.2, we observe that the differences mainly lie in the fine-grained details. We will work on improving the generating diversity.
M LIMITATION ANALYSIS
Our method shows a tendency to make more changes in small details rather than in large structures. We aim improve the diversity of generation in this regard. Additionally, method some-
18

---

Published as a conference paper at ICLR 2024
Imput
Iteration
Iteration 2
Iteration
Figure K.I: Inpainting results our PSM at different iterations. One-shot generation usually results in blurry content with unpleasing artifacts, while more iterations yield better results.
Input
Ours;
Ours²
Input
Ours;
Ours
Figure L.2: Visual examples of diverse generation for our method.
times struggles to understand objects when only few hints are given, illustrated by a few failure cases presented in Figure M.3 For instance, the missing part of the notebook is filled with the back- ground, and the recovered bus structure is incomplete. We attribute one of the reasons to the lack of high-level semantic understanding. We will further improve the generative capability of our model.
N ADDITIONAL QUALITATIVE COMPARISONS
We provide more visual examples on 512 x 512 Places2 Zhou et al. (2017) and CelebA- Karras (011)) Figures N.4 to N.8 Due space limit, we additionally add comparisons with CoModGAN Zhao et al. (2020) in Figure N.9 Compared to other methods, our method generates more photo-realistic and semantically consistent content. For example, our method successfully recovers human legs, airplane structures, and more realistic indoor and outdoor scenes.
Our model performs well when the input image contains sufficient visible pixel information, en- abling high-quality generation while maintaining coherence with the existing content. This success originates from our model's 's pixel spreading mechanism, which initiates from visible pixels and progressively diffuses valuable information throughout the image. This approach is particularly ef- fective for facial images characterized by strong inherent priors, such as symmetry and structural attributes. In the third example in Figure N.7, where the right half of the face is visible, model
19

---

Published as a conference paper at ICLR 2024
Input
PSM (Ours)
GT
Figure M.3: Failure cases of our PSM. It is difficult to recover the large-scale missing objects.
reconstructs a comparatively realistic and consistent result using visible features like eyes and beard, albeit with potential discrepancies in details like ears and nose. However, when the entire face is masked, as in the first example in Figure N.7, our generated facial image notably diverges from ground truth. This phenomenon also appears in natural scenes; for instance, in the third example in Figure N.4, our method successfully restores the airplane structure due to the visibility of the frontal section, while other methods fail. All the results demonstrate the effectiveness of method.
20

---

Published as a conference paper at ICLR 2024
Input
Original
Big-LaMa
MAT
LDM
PSM (Ours)
Figure N.4: Qualitative byde-b comparisons of state-of-the-art methods 512 512 Places2 dataset. Please zoom in for a better view. Our PSM produces structures and details that are more realistic and reasonable. Best viewed zoomed in.
21

---

Published as a conference paper at ICLR 2024
Input
Original
Big-LaMa
MAT
L.DM
PSM (Ours)
Figure N.5: Qualitative side-by bydeeb- comparisons of state-of- -re-thrtrrtt methods 512 x 512 Places2 dataset. Please zoom in for a better view. Our PSM produces structures and details that are more realistic and reasonable. Best viewed zoomed in.
22

---

Published as a conference paper at ICLR 2024
Input
Original
Big-I aMa
MAT
L.DM
PSM (Ours)
DOONIL
VOU DUOIDD
LIAPYIT LEINNTTR
Figure N.6: Qualitative side-by-side comparisons of state-of-the-ar methods 512 512 Places2 dataset. Please zoom in for a better view. Our PSM produces structures and details that are more realistic and reasonable. Best viewed zoomed in.
23

---

Published as a conference paper at ICLR 2024
Input
Original
MADF
CoMoIGAN
MAT
PSM (Ours)
e
Figure N.7: Qualitative syde-by-side comparisons of state-of-the-- methods on 512 x 512 CelebA- HQ dataset. Please zoom in for a better view. Our PSM produces face outlines and details that are more realistic and reasonable. Best viewed zoomed in.
24

---

Published as a conference paper at ICLR 2024
Input
Original
MADF
CoModGAN
MAT
PSM (Ous)
G
Figure N.8: Qualitative side-by-side comparisons of state-of-the-are methods 512 512 CelebA- HQ dataset. Please zoom in for a better view. Our PSM produces face outlines and details that are more realistic and reasonable. Best viewed zoomed in.
25

---

Published as a conference paper at ICLR 2024
Inpat
CoModGAN
(Ous)
Inpot
CoMoUGAN
(Sus)
Figure N.9: Qualitative comparisons between CoModGAN and our PSM 512 512 Places2 and CelebA-HQ datasets. Please zoom in for a better view. Our PSM produces structures and details that are more realistic and reasonable.
26


Reviews:
[{"review_id": "Wqq7ekxT2E", "reviewer": "Reviewer_HZRy", "summary": "This paper proposes a pixel spread model for image inpainting, which iteratively employs decoupled probabilistic modeling. Specifically, this work combines the optimization efficiency of GANs with the prediction tractability of probabilistic models and selectively spreads informative pixels throughout the image in a few iterations, improving the completion quality and efficiency. Experimental results show the proposed method achieves a state-of-the-art performance on several benchmarks.", "questions": "1. Please explain in detail how the uncertainty is obtained in each iteration.\n2. The third paragraph in Section 3.1.1 is inconsistent with Figure 2. The input image x_t at time t should be x_{t-1} at time t-1, right?\n3. Please discuss in detail the effect of the learnable function F in Eq. (6).\n4. The authors state several times that the proposed method improve the efficiency. Please demonstrate in detail, e.g., the comparison of inference time.\n5. What is the mask size in Table 4?\n6. In the experiments, please add the comparisons with more recent methods, e.g.,\n[a] Chu et al., Rethinking Fast Fourier Convolution in Image Inpainting, ICCV 2023.\n[b] Sargsyan et al., MI-GAN: A Simple Baseline for Image Inpainting on Mobile Devices, ICCV 2023.\n[c] Ko et al., Continuously Masked Transformer for Image Inpainting, ICCV 2023.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The proposed pixel spread model makes use of the merits of GANs’ efficient optimization and the tractability of probabilistic models.\n\nGood performance is achieved on several benchmark datasets.", "weaknesses": "There are several unclear statements listed as follows.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2nS5SZO3aC", "reviewer": "Reviewer_EPEv", "summary": "In this paper, the authors propose the pixel spread model (PSM) to gradually fill the masked regions of an image. Specifically, the model learns the mean by GAN loss and the variance by negative log likelihood (NLL). In each iteration, the model first makes the prediction, then picks the most valuable pixels to fill the region and leave the others masked. Experiments show that the method works well.", "questions": "Please see above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The idea is novel to gradually fill the regions through the computed mean and variance.\n- The method is efficient by utilizing the optimized computational resource of GAN model and the thoughts of autoregressive model.", "weaknesses": "- Some parts of the paper is not very clear. For example,\n    - What's the meaning of $y^i$ in equation (1)? Is it the groundtruth?\n    - More explanation is necessary for equation (5). What happens when $m_t=1$ and $m_0=0$ or vise versa?\n    - Is the uncertainty map computed only in the mask region or the entire image?\n    - Why $\\sigma_t$ is in the range $[0,1]$?\n\n- Comparison to Controlnet is necessary.\n- It seems that sometimes the model is hard to grasp some structure of the image. For example, in the last row of Fig.4, the result of Model A cannot reconstruct the structure of stairs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QzvocTUM3w", "reviewer": "Reviewer_N7bD", "summary": "Authors present an iterative model for image in-painting, aiming at\nfilling large-holes with realistic looking areas. The model uses a GAN\nand a Gaussian model to estimate an uncertainty term, which as far as\nI can understand aims to find a covariance matrix that will minimize\nnegative log-likelihood of real intensities with respect to a Gaussian\nmodel that uses the GAN output as the mean. Thus, this model uses the\ncovariance to account for the synthesis error. Pixels with low\nvariance are added to the current prediction and the process is\nrepeated until the entire empty region is filled. Experiments with\nlarge-scale data sets are given.", "questions": "1. The in-painting results are suspiciously good in some\n   cases. Can authors explain how their model can be so accurate - not\n   just realistic - in M7 and M8? May there be a problem with training\n   and test data leakage? \n\nI am happy to increase my score, however, first I would like to wait authors' response to this question.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. A very simple method that gives really good results.\n2. Authors experimented with large-scale data sets.\n3. Results are very impressive. The presented comparisons with\n   state-of-the-art methods shows that the proposed iterative scheme\n   yields better in-painting than all the rest. Furthermore, it does\n   it in a fraction of the time compared to the most recent methods. I\n   believe these results are extremely promising, and if they can be\n   reproduced they may have a substantial impact.", "weaknesses": "1. The training strategy is not well explained. Authors should clarify\n   whether and how the iterations are taken into account during the\n   training.\n2. There are some suspiciously accurate in-painting results in the\n   appendix - particularly in Fig. M7. Can authors explain how the\n   proposed model can be so accurate? I can understand being\n   realistic, however, the third column shows that the generation is\n   pretty much the same. Similar in-apinting results are given in\n   Figure M8.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lkg4WjBKkT", "reviewer": "Reviewer_rxeq", "summary": "In this paper, the authors propose a novel method to tackle high-resolution image inpainting problems using an iteratively decoupled probabilistic modeling approach. In particular, given a masked image, the proposed method iteratively inpaints masked pixels by a GAN model, as well as the variance of the generated pixel. By keeping the low variance pixels the masked area is reduced. Then this step is repeated until the entire masked area is filled. \n\nThe authors did comprehensive evaluations with other methods and showed that the proposed method achieves the best performance and is more computationally efficient.", "questions": "1. In Eq. 1, does $\\mathcal(N)$ mean the density function of the Gaussian distribution?\n2. In Eq. 1, what's the relationship between the $y$ being integrated and the $y$ in the integral bounds? Are they the same or different?\n3. In Eq. 1, where does $y_i$ come from during training? Is it the ground-truth unmasked image or the GAN-generated one? If it is the latter, it also depends on $\\theta$?\n4. Above Eq. 9, there seems to be a typo: 'filed' -> 'field'", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The originality of the proposed method mainly comes from the decoupling of mean and variance estimation for each pixel. In particular, at each iteration, the variance gives a metric to select the high-quality pixels as the input of the next iteration so that GAN artifacts are reduced. Also, since at each iteration, the entire image is generated, the whole process is scalable to high-resolution and large-hole inpainting problems. The experiments show strong numerical evidence of the performance of the method. Overall the paper is well-written and solid.", "weaknesses": "Overall the paper is well-written and sound. I have some specific questions regarding the math expressions (see the sections below).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)