Context-Aware Image Completion
Authors: Jinoh Cho
Minguk Kang
Vibhav Vineet
Jaesik Park
Venue: Submitted to ICLR 2023 (2023)
Keywords: Image Completion
Image Inpainting

Abstract: Image completion is a task that aims to fill in the missing region of a masked image with plausible contents. However, existing image completion methods tend to fill in the missing region with the surrounding texture instead of hallucinating a visual instance that is suitable in accordance with the context of the scene. In this work, we propose a novel image completion model, dubbed Refill, that hallucinates the missing instance that harmonizes well with - and thus preserves - the original context. Refill first adopts a transformer architecture that considers the types, locations of the visible instances, and the location of the missing region. Then, Refill completes the missing foreground and background semantic segmentation masks within the missing region, providing pixel-level semantic and structural guidance to generate missing contents with seamless boundaries. Finally, we condition the image synthesis blocks by using the completed segmentation mask to generate photo-realistic contents to fill out the missing region. Experimental results show the superiority of Refill over state-of-the-art image completion approaches on various natural images.

Under review as a conference paper at ICLR 2023
INSTANCE-AWARS IMAGE COMPLETION
Anonymous authors Paper under double-blind review
ABSTRACT
Image completion is task that aims to fill in the missing region of masked image with plausible contents. However, existing image completion methods tend to fill in the missing region with the surrounding texture instead of hallucinating a visual instance that is suitable in accordance with the context of the scene. In this work, we propose a novel image completion model, dubbed Refill, that hallucinates the missing instance that harmonizes well with and thus preserves- the original context. Refill first adopts transformer architecture that considers the types, locations of the visible instances, and the location of the missing region. Then, Refill completes the missing foreground and background semantic segmentation masks within the missing region, providing pixel-level semantic and structural guidance to generate missing contents with seamless boundaries. Finally, we condition the image synthesis blocks of Refill using the completed segmentation mask generate photo-realistic contents to fill out the missing region. Experimental results show the superiority of Refill over state-of-the-ar image completion approaches on various natural images.
INTRODUCTION
Image completion is the task of restoring the masked regions in an image, which requires understanding of the unmasked instances and the various relationships among them. Researchers have worked develop image completion models for practical applications, such image editing (Jo
& Park 2019; Ling et al. 2021). restoration (Wan et al.] 2020; Liang et al.) 2021), and object removal (Shetty et al. 2018). Most previous models, however, focus on filling the missing region realistically without considering the instance that needs to be restored. For example, we observe that even the cutting-edge image inpainting model (Li et al. 2022) tends to complete the missing region with surrounding textures rather than attempting to restore the lost instance; this limits the usage of image completion models in real-world applications.
Removal of focal instance in a scene can lead to substantial context change. For example, the removal of the horse the image Figure changes the local context around the missing region from "a person riding horse on the beach" "a boy walking the beach". HVITA (Qiu et al.) 2020) is the only work that tackles such substantial context change, which occurs from the complete removal of visual instance from the scene. However, HVTTA has three major limitations: (1) HVTTA mainly targets rectangle masks and thus lacks generalization to other mask forms, (2) the completed image produced by HVITA exhibits abrupt changes along the boundaries between the generated and original regions, and (3) HVITA has heavy reliance on refinement network to produce realistic images. To alleviate these issues, we propose a new framework called Refill that leverages a predicted semantic segmentation mask as guidance for image completion.
Refill performs image completion in three steps: 1) predicting the class of the missing instance, 2) generating a semantic segmentation mask of the missing region, and 3) completing the masked image using the segmentation guidance. Specifically, Refill predicts the class of the missing instance based on the context of the image, which is determined by mining the inter-instance co-occurrence using transformer network. Then, Refill generates the segmentation masks of both the missing instance and the background area of the missing region individually using conditional GAN and transformer body reconstruction network. Finally, by taking the generated segmentation mask as input, our framework generates context-friendly instance and its background, which fills in the masked image to finally produce a realistic natural image. The proposed context-awand oegmentation-uuide guided image completion framework enables Refill to handle missing regions with arbitrary shapes (such

---

Under review as a conference paper at ICLR 2023
Input
MAT
HVITA
Ours
Target
0.6416
0.5913
0.7620
Context Query: person riding horse on the beach" "
Figure I: From the first column, Input image with a missing region, results state-of-the-ar image completion approaches, such MAT (Li et al. 2022) and HVITA (Qiu et al. 2020), Our result (Refill), and the target image. We compute CLIPScore around the generated part using the query text. As approach generates horse complete the image rather than fills using background textures, CLIPScore of our result exhibits the best performance among the other models.
scribbles), unlike HVITA (Qiu et al. 2020) which is best suited for missing rectangular regions. Note that Refill avoids the need for refinement network, unlike HVTTA, which heavily relies on the performance of refinement network.
To evaluate and compare our model against existing methods, we first employ an off the-sseeff image captioning network, OFA (Wang et al. (022), to produce a caption for each missing region of masked images. We propose to use the produced caption as the context query, which represents the context of the missing regions. To measure how much the context of the image changes after completion, we propose to two evaluation metrics: (I) CLIPScore (Schönfeld et al., [02II) I). which employs CLIP visual and textual encoders to determine whether the generated image region is well aligned with the context query, and (2) Visual Grounding Accuracy (VGA), which uses pretrained visual grounding model (Wang et al. 2022) to determine whether the context query can successfully ground the generated image region. We also evaluate our method using conventional image quality assessment metrics, including FID (Heusel et al. 2017) and LPIPS (Zhang et al. 2018). On COCO- panoptia (Lin al. 2014)/Visual Genome (Krishna al. 2017) datasets, Refill shows comparable visual quality 1DD==7.84/5.84. to the state-of-the-art image completion approach such MAT (Li et 2022), while Visual Grounding Accuracy and CLIPScore are 12.472/ 472/14 10.1% and 0..27/00.02 better than HVITA. These results demonstrate that our approach can complete the missing regions of masked images in context- fantext-rriedly manner to yield high fighquuaity images.
Our contributions are summarized as follows:
We propose novel framework called Refill which completes the missing region of masked images in context-friee friendllfredlyy manner, preserving the original context by leveraging a seg- mentation mask to encourage visual consistency between the generated and unmasked areas without relying on refinement network.
We present novel combination of two transformer-basee modules which facilitates our context-aware image completion pipeline. The missing instance inference transformer predicts the class of the missing instance effectively. The transformer-boue background segmentation completion network shows netter-recoverce segmentation masks, especially under the presence of large missing regions.
We propose to adopt CLIPScore and VGA to evaluate the context consistency between the original image and the completed image.
Refill produces new visual instances in missing regions that are visually consistent with the unmasked areas. Refill also shows better performance in CLIPScore and VGA metrics compared to the baselines and exhibits comparable FID performance compared to the state-of-the-art approaches.
2

---

Under review as a conference paper at ICLR 2023
2 RELATED WORK
Traditionally, image completion task has been solved using approaches based on diffusion- -ased
methods (Bertalmio et al. 2000 Ballester et al. 2001) and patch- based methods CCriminisi et al.
2003: 2004; Ding et | 2018 Hays & Efros] 2007: Meur et al., 2011 Lee et al.) 2016 Sun et al.
2005). The fundamental assumption of these models is that the missing parts of an image can be found and replaced with the remaining regions, making the model complete the missing region with
only low-level features and repetitive patterns in the image.
With the advancement of deep learning, image completion models based on deep generative models have become the mainstream of photo-realistic image completion. Context encoder (Pathak et al. 2016) benefits from adversarial training inspired by Generative Adversarial Network (GAN) (Good fellow et al.] (2014) and shows perceptually more plausible results. VQGAN (Esser et al., 2021
and Palette (Saharia et al. 2022) introduce more advanced generative models: auto-regressive and denoising diffusion probabilistic modeling, and the approaches show compelling outputs. Along with efforts to introduce better generative models, there have heen studies to improve the completion performance by revising existing architectures and convolution operations. [Song et al. 8a); Yan
et al. (201 8); Yu et al. 2018); ; Liu et al. (2019) propose contextual attention layers for long-range contextual encoding and image completion. Liu al. (2018) propose partial convolution layer to alleviate color discrepancy and blurriness in completed images. Yu et al. (2019) generalizes the partial convolution by introducing dynamic feature selection mechanism at each spatial coordinate. Recent works tackle orge--cale missing regions in masked images, more challenging setting. (Zhao ct al. (202I) propose a modulation technique to handle this setting. Li et al. (2022) devises new transformer blocks to take advantage of long-rang raage context interaction to complete images.
Generating photo-realistic images from semantic segmentation masks is also relevant work for image completion. The main goal is to condition the segmentation mask while preserving its spatial semantic information. To solve this issue, the authors of SPADE (Park et al. 2019) propose a opatially-adaptii (de)normalization layer by modulating pixel-wise features. Moreover, OASIS (Schönfeld al. 2021) strengthens the discriminator by replacing conventional real or fake discriminator with a segmentation- based architecture. This mechanism improves the generator to synthesize realistic images well aligned with semantic layout. These works are originally not intended to complete masked images but designed to generate photo-realistic images from the noise vector conditioned on semantic segmentation masks. We bring SPADE and OASIS and give segmentation mask guidance for the missing region completion.
Other works employ semantic/structural information rather than directly complete the corrupted images. EdgeConnect (Nazeri et al.) 2019) and E-CE (Liao et al. 2018) propose models that recover edge details first then fill out color next. Beyond using structural information, SPG-Net (Song et al. 2018b) and SG-Net (Liao et al.] (020) use semantic segmentation maps to contain more informative guidance than just edge. Our approach is close to SPG-Net and SG-Net, in that we recover the segmentation masks and generate the content. However, our method differs in that we handle more challenging scenarios where the instances are entirely removed. To best knowledge, the only work for the context-aware image completion is HVITA (Qiu et al.] 2020), where a target instance wholly removed from an image. HVITA consists of four steps: (1) detecting visible instances, (2) constructing a graph using detected instances to understand the scene context, (3) generating missing instance and placing it the missing region, and (4) refining the inserted image. Yet, HVITA is the best suited for rectangular mask and show poor image quality. especially around the boundaries. contrast, we are free handle masks with arbitrary shapes since we directly generate new instances within missing regions guided by completed segmentation map, unlike HVITA. The completed segmentation map helps to understand the context of images and encourages visual continuity on the boundaries between generated and unmasked regions. It also avoids relying on the refinement networks that is used in HVITA.
METHOD
Given an uncorrupted image I, create corrupted image Iy using binary mask M. The corrupted image can be expressed as IM = I @ M, where @ is element-wise multiplication. Our framework aims complete the corrupted image IM, where the visual instance is completely
3

---

Under review as a conference paper at ICLR 2023
Step
Step
Step 3
lsput
Instanee Segminntation Mask
Sogmentation Guided Image Completion Network
Noise Noo,
Instance Segmentation
Generator
Missing Instance Inference Transformer
DETR
Background Segmentation
Completion
Netwark
using BROX
Cuiput
Segmentttion Mask
Segmentizion
Figure 2: Overview of the proposed approach, called Refill. Refill completes the image in three steps: (1) infer the missing instance class, (2) complete a segmentation map in the missing region, and (3) translate the segmentation map image to hallucinate the missing region.
removed. This problem is challenging since the model should generate a target instance. At the same time, the generated instance should keep visual continuity with the existing parts.
Our framework (Refill) completes masked images in three steps. First, we infer a contextually appropriate instance by figuring out the category of the missing instance (Sec. 3.1) Second, complete the missing semantic segmentation map based on the inference result from the previous step (Sec. 3.2h. Finally, we transform the masked image with semantic segmentation into realistic color image (Sec. 3.3) The framework enables each module to concentrate on the individually assigned task (instance inference, missing segmentation map completion, and image generation) instead of doing these in one shot. This is the key difference from most existing image completion approaches that mainly focus on the realism completion results do not consider semantics or which instance should be painted there. The overview figure of our framework is shown in Figure
3.1 MISSING INSTANCE PREDICTION
To obtain the relationship information between instances in given scene, Refill first predicts instance bounding box coordinates, instance classes, and a semantic segmentation map using the pre-trained DETR (Carion et al. 2020). Let the panoptic segmentation map SM DETR( MM) then can extract box coordinates of the visible instances B [b,,- bu] and object classes c = [ci, cx] from SM. where & is the number of predicted instances. Then, infer the class of the missing instance ytargets Refill inputs the predicted classes of the visible instances along with missing region token into a transformer network, called missing instance inference transformer. We first convert the visible instances' classes into learnable input tokens using single linear layer. A quick approach is utilize object queries from DETR as input tokens directly, but we observe such method showed worse performance compared to employing new learnable class embeddings. Furthermore, to inject the location information of the visible instances, we convert their bounding box coordinates into positional encoding vectors and sum them to the learnable class embeddings. To acquire the positional encoding vectors, we input the normalized center coordinate (Cz, Cy), width (W), and height (H) of the bounding box single linear layer where the activation is sigmoid function. We also apply the same procedure for the missing region token, creating an embedding for missing region also explore different ways develop the positional encoding vectors and
estimation. In addition, show that the adopted positional embedding mechanism provides the best missing instance estimation performance in Sec. 4....] Below formulations are mathematical expressions of how our missing instance infer transformer works.
2o Ectass + Epos MLP(C') (MLP(B')),
Zp E R (k+ljxd
(1)
z ())(L 21-1; z MLP(LN(=1)) 2/1 y LN(2)))
= ...y
(2)
= 1 L
(3)
(4)
4

---

Under review as a conference paper at ICLR 2023
where MLP is multi perceptron, is sigmoid activation, d is the dimension of embedding vectors, B' = -bo) U B where by is the missing region bounding box coordinate, and c' [Cu] U where cp is null class for missing region classification. The missing instance inference transformer consists of 12 transformer encoder layers (I. = 12) with 8 heads. The missing region token interacts with the visible region tokens via the self-attention mechanism in the network. Thus, the network can predict the plausible class of the missing instance based on the detected instances and their location information. More details are in Appendix A.3.1
3.2 SEMANTIC SEGMENTATION MAP GENERATION
Utilizing the predicted class of the missing instance from the previous step, we aim to generate the semantic segmentation map of the missing region. We create the segmentation map of both the instance and the background area individually with separate modules (instance segmentation generator and background segmentation completion network) and obtain the final segmentation map by inserting the missing instance segmentation into the background segmentation as shown in Figure 2
We perform the missing instance segmentation generation using two modules: generator and discrim- inator, for the instance segmentation. The segmentation completion net aims create a plausible segmentation map corresponding to the predicted instance class. For the implementation, we use the architecture from BigGAN (Brock al. 2019), one the most successful conditional GANs, with slight modification. We input the predicted missing instance class from the previous step and the box coordinates of the missing region to the Conditional Batch Normalization (De Vries) ct al. 2017) module in the instance segmentation generator. We train the model by using spectral normalization (Miyato et al. 2018) and hinge loss (Lim & Ye 2017).
Second, the background segmentation completion network produces the segmentation map of the remaining region without attempting to generate the missing instance. To do this, we randomly scribble the ground truth segmentation map and let the background segmentation completion network restore it using cross-entropy loss. We experimentally found that this procedure can reconstruct background segmentation maps successfully. This module consists of convolution heads and tails with a transformer body. We explore that the transformer body helps to reconstruct the missing background segmentation map, especially large hole setting (See Sec. 4.5.2). Finally, we obtain the overall segmentation map of the missing region by inserting the instance segmentation into the background segmentation. More details are Appendix/A.33 A.3.3
3.3 SEGMENTATION IMAGE COMPLETION
Using the predicted missing region segmentation map as guidance, a UNet (Ronneberger et al. 2015)-like completion model reconstructs the missing region of the masked image. The model takes in the masked image as input and outputs the restored version where SPADE (Park et al. 2019)/ OASIS (Schönfeld et al., (202]) blocks force the model reflect the semantics of the segmentation map. The details about the hyperparameters, objective function, and the model architecture are available in Appendix A.3.3] Since Refill can complete masked image using any segmentation map, we can reconstruct diverse target instances of the same class by feeding different segmentation maps into the completion model. Figure 4 shows an example.
4 EXPERIMENT
4.1 DATASETS
We two datasets called COCO-pane -anopttt (Lin et al., 2014) and Visual Genome (Lin et al. 2014). which HVITA (Qiu et al. 2020) adopted to evaluate performance.
COCO-panoptie COCO-panoptic (Lin et al. 2014) dataset contains 118K images for the train set and 5K images for the validation set with 80 things classes and 91 stuff classes. The dataset consists of natural images with multiple instances and has been hardly studied than center-aligned datasets (c.g., FFHQ (Karras et al.) 2019), CelebA-HO (Liu et al.) 2015), and ImageNet (Deng et al. 2009) terms of image completion. For the experiment, create missing region using
5

---

Under review as a conference paper at ICLR 2023
two procedures: (1) first selecting missing rectangular region that contains instance, then (2) randomly cropping the image to be from 10% to 50% of the whole image, including the missing rectangular region. We randomly picked 50 points around the bounding box and drew thick lines between those points to make irregular regions. We select 30 classes from the things classes that are frequently observed in the images for the demonstration.
Visual Genome. We adopt Visual Genome (Krishna et al. 2017) to check the generalization ability of the proposed approach (Refill). The dataset contains 1 images with fine-grained 34k object categories. Due to the category mismatch between COCO-pane COCO-panoptic and Visual Genome, we use the DETR- preedccrdd results as an annotation. Then, we make missing regions using the detected object boxes with the same size limitation used for creating the COCO-panope dataset.
4.2 BASELINE
HVITA QQu et al. 2020) is closely related work considering our problem setting. HVITA hallucinates the visual instance in three steps. First, the detection module detects visible instances, and the graph module predicts the features of a missing instance. Second, a conditional GAN generates a square- shaped RGB visual instance by conditioning the missing instance features and inserting them into the missing region. Finally, the refinement network improves visual continuity between the inserted part and the existing part. However, HVITA is originally designed to handle a rectangular mask and the boundaries due to the object insertion step, shown in Figure 3 We could not find any publicly available code, so we implemented HVITA carefully by referring to the details in the paper (Qiu al. 2020).
still suffers from abrupt discontinuity
MAT (Li et al. 2022) is one of the cutting-edge image completion models built on GAN framework. It consists of newly designed transformer blocks with a style manipulation module. We use the official author's 's implementation for training and evaluation.
4.3 METRICS
LPIPS and FID. There are several metrics for image quality measurement. LI, MSE, and PSNR quantify a pixel-wise error, and SSIM calculates the similarity between a generated image and the original one using luminance, contrast, and structure information. These metrics, however, are known be inconsistent with human perception (Zhang et al., 2018). In addition, since there are various possible answers to complete the missing region, LI, MSE, and PSNR may give high value, although completion results are plausible. To overcome this issue, we use LPIPS (Zhang et al.) 2018), which is well known to agree with human perception by computing L2 distance learned feature space (ImageNet-trainee) VGG). Moreover, we utilize Fréchet Inception Distance (FID) (Heusel et al. 2017 to assess the realism of completed images.
CLIPScore. To assess whether the hallucinated instance is perceptually well- well-aligned with the given text description, we employ CLIPScore computed using pretrained CLIP ViT-B/32 (Radford et al.)
2021 visual and textual encoders. CLIPScore computes the cosine similarity between hallucinated instance embedding and the query text embedding. For the given hallucinated instance embedding h and text description embedding t, we can compute CLIPScore( h, t) = 2.5 max(cos(A h. t), 0). This formulation is originally devised to evaluate image captioning performance in a reference-free manner (Hessel et al. 202T), but we use it to assess the perceptual alignment of hallucinated instances or entire generated images.
In contrast to LPIPS, which trims feature space using a uni-modal dataset with cross-entropy loss, CLIP organizes its feature space by optimizing Info-NCE loss (Sohn) 2016) on cross-modal dataset made up of 400M (image and caption) pairs. Specifically, while the VGG backbone used for computing LPIPS gets supervision from pre-defined ImageNet classes Deng et al. (2009). CLIP encoders get strong feedback from vast amounts of natural images and texts on the internet. As the CLIP encoders show the capability of producing general representations via their superior zero-shot transfer ability (Radford et al. 2021), they can extract distinguishable features even in unseen classes.
Visual Grounding Accuracy (VGA). Locating the bounding box for a given query sentence in the image is the goal of visual grounding. If the model hallucinates plausible instance similar to the original one, visual grounding model can place a bounding box around the hallucinated image given the text. For the evaluation, adopt OFA, grownding (Wang et al. 2022) trained Visual

---

Under review as a conference paper at ICLR 2023
Table I: Comparison image synthesis quality on COCO-panoptic and Visual Genome datasets. All models trained/finetuned on COCO-panople and evaluated on COCO-panople and Visual Genome (Zero-shot) images. We compare the quality of generated images using metrics: CLIPScore, Maices365-Standall (Zhou al. 2017) that contains 8M images and is finetuned with our modified COCO dataset. MAT and the other models trained from scratch only using the modified COCO. We mark the best, the second-best in
VGA, LPIPS and FID. MAT pre is pretrained
normal yellow and light yellow respectively.
COCO-panoptic
Visual Genome (Zero-shot)
Metric
CLIPseore
VGA (%)
LPIPS 1
FID
CLIPscore f VGA (%) f LPIPS 4 FID
MAT (Li cl 2022
0.614
3.320
0.087
7.192
0.615
3.237
0.087
6.488
MAT MATarveiel acreitee CL al 2022
0.606
3.216
0.093
7.895
0,608
3.141
0.093
7.302
HVITA (Quu al. 2020
0.567
17.873
0.132
10.496
0,568
18.606
0.129
9,311
Refillapude
0.626
26.107
0.122
8.519
0.628
28.181
0,120
6.658
Refillonci
0.641
30.345
0.119
7.284
0.644
32.713
0.116
5,849
Input
MAT
HVITA
Target
DETR
Our Seg.
Target Sog.
Figure 3: From 1st column to 5th column: Input, MAT (Li et al. 2022 HVTTA (Qu et al. 2020). Ours, GT. From 6th column last column: DETR (Carion al.) 2020) seg, Our Seg and GT Seg. Refill restore the segmentation as shown in 6th column. We leave out more results on the Appendix A.4
Genome Captions (Kishha et al.) 2017). RefCOCO (Mao et al. 2016), and variants of RefCOCO, such as RefCOCO+ and ReiCOCOg. The purpose of using VGA is similar to CLIPScore, core difference is the way how the feature space is annealed. The visual grounding model is trained using the O. 1M scale datasets, which is much smaller than the dataset used for CLIP, but the dataset contains high-quality and dense annotations.
the
Due to the lack of ground truth text descriptions for CLIPScore and VGA, we generate plausible sentences using pretrained grounded image captioning network from OFAcaptioning (Wang et al. 2022 We measure VGA by setting mloU threshold as 0.5.
4.4 EVALUATION RESULTS
To show the strengths of Refill, we perform the image completion task on the modified COCO -anoopiic dataset (Sec. 4.1). We train MAT pre (Li et al.) 2022), MAT (Li et al. 2022), HVITA QQu et al. 2020), and our model using the dataset and evaluate the models using a test-split of COCO- dataset. The results are summarized in Table I MAT pre shows the lowest LPIPS and FID scores of 0.087 and 7919,, respectively. But it exhibits inferior results CLIPscore and VGA on which Refill gives the best results (CLiPscore is 0.641, and VGA is 30.345%). In the case of HVITA, the most relevant baseline with our model. it can generate the target instance better than MAT pre and MATscrateh- scratch- Still, the generated instances are less realistic than the other models, seen in Table I and Figure Refill is the model that compensates for the shortcomings HVITA but reinforces the strengths. Refill attains comparable FID of 7.284 with the number of MAT pre (7.192). At the same
7

---

Under review as a conference paper at ICLR 2023
Input
Ours
Ours2
Ours3
Target
Figure 4: Example of diverse completion results of the proposed approach.
Input
Ours
Target
Input
Ours
Target
Figure 5: Failure cases. The proposed approach may fail to predict the same class of the target image, which may harm the evaluation scores. However, the generated images still follow the scene context. There are more failure cases in the Appendix A.4
time, Refill yields 0.027 and 12.472% better CLIPscore and VGA than the second best models. Note that MAT pre is trained on Places365-Standant which is 80 times larger than the COCO-panoptic. The scale of the training data causes unfair evaluation between our model and MAT pre. and the comparison with MATseratch indicates better performance of Refill among the other completion models adopted in our experiments.
To test the generalization ability of each model, we evaluate the on Visual Genome the zero-shot fashion. can be seen in Table I Refill proves the excellence in FID, CLIPscore, and VGA metrics except for LPIPS. Note that high LPIPS does not necessarily indicate low-fidelity results, where the results may include instances that differ from those of the and
completion models
original images as shown in Figure 3 5) Since LPIPS uses pixel-wise evaluation in the feature space, the model must generate an inpainting similar to the original image to obtain low LPIPS score. However, our model completes the missing regions with visual instances of various shapes and classes, which are likely to differ from the original instance. Such diverse outputs result in comparatively higher LPIPS scores.
4.5
ABLATION STUDY
4.5.1
POSITIONAL ENCODING FOR MISSING-INSTANCE PREDICTION
Since instances' location information as important as the class of instances to understand the scene, we devise six posi- tional encoding variants. Let's denote relative bounding box coordinate follow: Rs C22 M2 and R. C, Myy where (Mz, M2) and (Ca, are missing region center coor- dinate and detected instances' bounding box center coordinate. respectively. H and W indicate the width and height of the bounding box. All the variables mentioned above are normal- ized to [0, 1j. ABS4C represents using Cz, Cy, H, W. ABS2C only uses C₂ and Cy. RELAC represents using R2: Ry: and W. REL2C only uses Rg, Ry. No PE represents not using any positional encoding methods. Learnable means using the learnable positional encoding method. GCN indicates graph classification module in HVITA (Qiu et al. 2020).
Cosstoottton Accracy PM
69
13
Lamoba
Figure 6: Comparison of the 6 posi- tional encoding variants in missing instance infer transformer and GCN module in HVTTA (Qiu et al. 2020) on COCO- COCO-panoptic
Among six methods, using the ABS4C method shows the best performance. Also, missing instance inference transformer with ABS4C positional encoding shows 4.5% increase in classification accuracy performance compared to the No PE model. Moreover, using transformer with ABS4C
8

---

Under review as a conference paper at ICLR 2023
Input
Conv
Transformer
Target
0.75
Conv-body
Transformer body
mlou
0.65
0.55
0.45
0%
25%
50%
75%
100%
Hole Size
Figure 7: Predicting semantic masks given the severe missing regions. Transformer-basse background completion network is more robust the hole size than the Convolution-basee version that often shows unwanted artifacts, as highlighted in yellow dotted boxes. The hole sizes of the first and the second row examples are 76.4% and 71.3%. respectively.
positional encoding significantly improves the performance of the GCN module in HVTTA 17%. See the details of six variants and their performance comparison in Figure
4.5.2 BACKGROUND COMPLETION NETWORK
Inspired by MAT et al. 2022), which uses a transformer architecture the body of the network to complete large-scale masked images, also adopt the transformer architecture in the body of the background segmentation completion network. Here, we use the Transformer body to complete missing background segmentation, not image. As shown in Figure transformer architecture helps reconstruct background semantic labels, especially for the large hole in the segmentation. As the hole size segmentation from DETR increases, the mloU performance gap between Transformer- and Conv-Based background segmentation completion network becomes more prominent. We also find that the Conv-based one generates artifacts on missing background segmentation in large hole settings, as shown in Figure 7 This demonstrates that Transformer architecture is better for recovering missing segmentation by leveraging global-range context interaction in large hole setting. Both networks used the same number of parameters (44M) and were trained from scratch for this experiment.
4.6 FAILURE CASES
Refill may fail to generate correct class instances due to the wrong predicted class from the missing instance inference transformer. However, shown in Figure the generated instances well harmonized with the remaining parts. In particular, Refill generates cup and a fire hydrant instead of cake and horse. The images still convincing.
CONCLUSION
This paper presents novel framework called Refill for image completion, which hallucinates context- friendly visual instances instead of filling the missing region with surrounding textures. Through extensive experiments, we demonstrate the superiority of Refill in terms of CLIPScore and VGA metrics while achieving comparable FID scores over the baseline models. This work could be extended to more general scenarios, such image completion for images with multiple missing regions or hallucinating multiple context-friendly instances within a single missing region.
Limitation The generation quality of our approach heavily depends semantic image synthesis blocks (SPADE and OASIS), Refill inherits their shortcomings, especially generation diversity, as can be seen in Figure Moreover, Refill is not aware of "no instance" and high-order geometry such as occlusion order and instance's 's orientation. Our approach demonstrated with the 30 classes of COCO- -anoopii things classes. Rather than using pre-defined categories, open-set segmentation and text-to-image translation would be intresting approaches. Solving these limitations could be interesting future works.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Coloma Ballester, Marcelo Bertalmio, Vicent Caselles, Guillermo Sapiro, and Joan Verdera. Filling-in by joint interpolation of vector fields and gray levels. IEEE transactions on image processing. 2001.
Marcelo Bertalmio, Guillermo Sapiro, Vincent Caselles, and Coloma Ballester. Image inpainting. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, 2000.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large Scale GAN Training for High Fidelity Natural Image Synthesis. In Int. Conf. Learn. Represent, 2019.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey 7agoruyko. End-to-end object detection with transformers. In Eur. Conf. Comput. Vis., 2020.
Antonio Criminisi, Patrick Perez, and Kentaro Toyama. Object removal by exemplar- esemplar-based inpainting. In IEEE Conf. Compul. Vis. Puttern Recog., 2003.
Antonio Criminisi, Patrick Pérez, and Kentaro Toyama. Region filling and object removal by exemplar-based image inpainting. IEEE Trans. Image Process. 2004.
Harm De Vries, Florian Strub, Jérémie Mary, Hugo Larochelle, Olivier Pietquin, and Aaron C Courville. Modulating early visual processing by language. In Adv. Neural Inform. Process. Syst., 2017.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conf. Comput. Vis. Pattern Recog, 2009.
Ding Ding, Sundaresh Ram, and Jeffrey Rodríguez. Image inpainting using nonlocal texture matching and nonlinear filtering. TIP, 2018.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In IEEE Conf. Comput. Vis. Pattern Recog., 2021.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems (NeurIPS), 2014.
James Hays and Alexei Efros. Scene completion using millions of photographs. TOG, 2007.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference- free evaluation metric for image captioning. In IEEE Trans. Circuit Syst. Video Technol., 2021.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Adv. Neural Inform. Process. Syst., 2017.
Youngjoo Jo and Jongyoul Park. Sc-fegan: Face editing generative adversarial network with user's sketch and color. In Int. Conf. Comput. Vis., 2019.
Minguk Kang, Joonghyuk Shin, and Jaesik Park. StudioGAN: Taxonomy and Benchmark of GANs for Image Synthesis. arXiv preprint arXiv:2206. 09479, 2022.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. arXiv preprint arXiv 14/2.6980, 2015.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li. David Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. Int. J. Comput. Vis., 2017.
10

---

Under review as a conference paper at ICLR 2023
Olivier Le Meur, Josselin Gautier, and Christine Guillemot. Examplar- based inpainting based on local geometry. In JEEE Int. Conf. Image Process., 2011.
Joo Ho Lee, Inchang Choi, and Min H Kim. Laplacian patch-based image synthesis. In IEEE Conf. Comput. Vis. Puttern Recog. 2016.
Wenbo Li, Zhe Lin, Kun Zhou, Lu Qi, Yi Wang, and Jiaya Jia. Mat: Mask-aware transformer for large hole image inpainting. In IEEE Conf. Comput. Vis. Pattern Recog., 2022.
Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In Int. Conf. Comput. Vis., 2021.
Liang Liao, Ruimin Hu, Jing Xiao, and Zhongyuan Wang. Edge-aware context encoder for image inpainting. In 2018 IEEE International conference acoustics, speech and signal processing (ICASSP), pp. 3156-3100 3160. IEEE, 2018.
Liang Liao, Jing Xiao, Zheng Wang, Chia-Wen Lin, and Shin ichi Satoh. Guidance and evaluation: Semantic-aware image inpainting for mixed scenes. Eur. Conf. Comput. Vis., 2020.
Jae Hyun Lim and Jong Chul Ye. Geometric GAN. arXiv preprint arXiv 1705.02894, 2017.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Eur. Conf. Comput. Vis., 2014.
Huan Ling, Karsten Kreis, Daiqing Li, Scung Wook Kim, Antonio Torralba, and Sanja Fidler. EditGAN: High-Precision Semantic Image Editing. In Adv. Neural Inform. Process. Syst. 2021.
Guilin Liu, Fitsum Reda, Kevin Shih, Ting-Chun Wang, Andrew Tao, and Bryan Catanzaro. Image inpainting for irregular holes using partial convolutions. In Eur. Conf. Compur. Vis., 2018.
Hongyu Liu, Bin Jiang, Yi Xiao, and Chao Yang. Coherent semantic attention for image inpainting. In Int. Conf. Comput. Vis., 2019.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Int. Conf. Comput. Vis., 2015.
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In IEEE Conf. Comput. Vis. Puttern Recog., 2016.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral Normalization for Generative Adversarial Networks. In Int. Conf. Learn. Represent., 2018.
Kamyar Nazeri, Eric Ng, Tony Joseph, Faisal Qureshi, and Mehran Ebrahimi. EdgeConnect: Structure Guided Image Inpainting using Edge Prediction. In The IEEE International Conference on Computer Vision (ICCV) Workshops, 2019.
Taesung Park, Ming- Yu Liu, Ting-Chun Wang, and Jun Yan Zhu. Semantic image synthesis with spatially-adaptive normalization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vitito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, Hfig--Peformance Deep Learning Library. Adv. Neural Inform. Process. Syst., pp. 8024-8035 2019.
Deepak Pathak. Philipp Krähenbühl, Jeff Donahue, Trevor Darrell, and Alexei A. Efros. Context Encoders: Feature Learning by Inpainting. In IEEE Conf. Comput. Vis. Pattern Recog. 2016.
Jiayan Qiu, Yiding Yang, Xinchao Wang, and Dacheng Tao. Hallucinating visual instances total absentia. In Eur. Conf. Comput. Vis., 2020.
11

---

Under review as a conference paper at ICLR 2023
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Int. Conf. Machine. Learning., 2021.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer- assisted intervention, 2015.
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In ACM SIGGRAPII 2022 Conference Proceedings, 2022.
Edgar Schönfeld, Vadim Sushko, Dan Zhang, Juergen Gall, Bernt Schiele, and Anna Khoreva. You only need adversarial supervision for semantic image synthesis. In Int. Conf. Learn. Represent., 2021.
Rakshith R Shetty, Mario Fritz, and Bernt Schiele. Adversarial scene editing: Automatic object removal from weak supervision. Adv. Neural Inform. Process. Syst. 2018.
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. In Adv. Neural Inform. Process. Syst., 2016.
Yuhang Song, Chao Yang, Zhe Lin, Xiaofeng Liu, Qin Huang, Hao Li, and C-C Jay Kuo. Contextual- based image inpainting: Infer, match, and translate. In Eur. Conf. Comput. Vis., 2018a.
Yuhang Song, Chao Yang, Yeji Shen, Peng Wang, Qin Huang, and C-C Jay Kuo. Spg-net: Seg- mentation prediction and guidance network for image inpainting. In Brit. Mach. Vis. Conf.. 2018b.
Jian Sun, Lu Yuan, Jiaya Jia, and Heung-Yeung Shum. Image completion with structure propagation. In ACM SIGGRAPH 2005 Papers, 2005.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Adv. Neural Inform. Process. Syst., 2017.
Ziyu Wan, Bo Zhang, Dongdong Chen, Pan Zhang, Dong Chen, Jing Liao, and Fang Wen. Bringing old photos back to life. In IEEE Conf. Comput. Vis. Pattern Recog., 2020.
Peng Wang, An Yang. Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Unifying architectures, tasks, and modalities through a simple tegueecceto-scquence learning framework. In Int. Conf. Machine. Learning.. 2022.
Zhaoyi Yan, Xiaoming Li, Mu Li, Wangmeng Zuo, and Shiguang Shan. Shift-net: Image inpainting via deep feature rearrangement. In Eur. Conf. Comput. Vis., 2018
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Generative image inpainting with contextual attention. In IEEE Conf. Comput. Vis. Pattern Recog., 2018.
Jiahui Yu, Zhe L. Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S. Huang. Free-Form Image Inpainting With Gated Convolution. In Int. Conf. Comput. Vis., 2019,
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In IEEE Conf. Comput. Vis. Pattern Recog., 2018.
Shengyu Zhao, Jonathan Cui, Yilun Sheng, Yue Dong, Xiao Liang, Eric I Chang. and Yan Xu. Large scale image completion via co-modulated generative adversarial networks. In Int. Conf. Learn. Represent., 2021.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions pattern analysis and machine intelligence, 2017.
12

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 ADDITIONAL EXPERIMENTS ON THE PROPERTIES OF MISSING INSTANCE INFERENCE TRANSFORMER
In this section, we perform diverse experiments to identify the effectiveness and the generalizability of the proposed missing instance prediction module. We first visualize the intermediate self-attention layers to verify the dependency between surrounding instances and the predicted instance. As shown in Figure 8 when our module predicts a missing class, the module gives more weight to the semantically related surroundings. We then perform another experiment to check whether module's 's prediction also depends on the position of the missing region. Figure g describes how predicted classes change over different missing regions. From these two observations, empirically conclude that our module rationally produces missing instance's class depending on the category and position of surrounding instances.
For the generalizability of our missing instance inference transformer, we adopt zero-shot setting and see if our module trained on COCO-panoptic dataset succeeds to produce accurate prediction class label of the unseen dataset, Visual Genome. Experiment results in Table show that prediction accuracies unseen dataset matches that of the seen dataset regardless of the choice in bounding box representation. Thus, we conclude our missing instance inference transformer well generalizes to unseen datasets as long as they share same class labels with the training set.
A.2 ADDITIONAL. EXPERIMENTS ON THE OBJECT REMOVAL. TASK
While our work primarily focuses on generating a plausible instance in the missing region, the task of object removal, which focuses overwriting the region with natural background, is also widely studied in the field of image inpainting. Here, we perform experiments to verify if our model is capable of object removal through small change in the pipeline. From the main framework Figure we omit pipeline for predicting and generating segmentation mask in Step 2 and use only background segmentation completion network. Experiment results from Table suggest that object "removed" images look as realistic as instance "inserted" images which are generated according to original framework this paper. Qualitative results can be found Figure 10
Missing instance infer transformer performance on COCO-pamoptic and VG (Zero-shot). The explanation of each methods including ABS4C, RELAC, ABS2C, REL,2C, Learnable, No PE and GCN are described in Sec. 4.5.1
Table 2:
COCO-panoptic
Visual Genome (Zero-shot)
ABS4C
67.548
67.236 (L 0.312)
RELAC
66.466
66.283 (} 0.133)
ABS2C
65.204
66.055 (1 1.149)
REL2C
65,865
66.370 (f 0.505)
Learnable
62,981
65.701 (1 2.720)
No PE
63.041
65.047 (1 2.006)
GCN
50.661
49.178 ( 1.483)
13

---

Under review as a conference paper at ICLR 2023
Wall other-
Table-
Laptop
0133
Layer 1
0375
0390
Layer 4
Layer 7
Mouse
Layer
Grass- merged
Tree- merged
Banner Person
Frisbee
0289
Layer
0200
Layer
0330
0200
Layer
Dog
0480
0193
Layer 10
0.19
Figure 8: Self-attention visualization from the missing instance inference transformer. We visualize the self-attention of the missing instance token in the intermediate layer (Layer 1, Layer 4, Layer 7 and Layer 10) of the missing instance inference transformer. The missing region of each images denoted by semi-transparent red mask.
Cup
TV
Bowl
Broccoli
Keyboard
Figure 9: Missing instance class prediction changes depending the position of each missing region. Each missing region the images is denoted by different colors of masks. Predicted classes each missing region are indicated with arrow.
14

---

Under review as a conference paper at ICLR 2023
Table 3: FID comparison between vanilla model in Refill and "no instance" version proposed in Appendix A.2 The vanilla model generates context-friente visual instances in the missing region while the "no instance" version fills the missing region with surrounding textures by simply skipping the foreground instance prediction.
Dataset
vanilla
"no instance"
COCO-panoptic
7.284
7.874 (f 0.590)
Visual Genome (Zero-shot)
5.849
6.887 (f 1.038)
Input
Ours
Original
STOP
Figure 10: Qualitative results of Refill's "no instance" version.
A.3 NETWORK ARCHITECTURES AND TRAINING DETAILS
We implement our completion model for experiments using PyTorch (Paszke al. 2019) library. We use PyTorch functions and define some necessary notations here to provide the architectural details of Refill.
EMBEDDING (dimout) indicates PyTorch embedding function. vanilla transformer encoder layer. FC (diminput, output) is single linear layer. CoNY (kernelsizes stride, padding) convolution layer. DBLOCK (channelsn, channelout, downsampling) and BIGG- BLOCK (channeli.. channelout, upsamplingg dim dimsshared) are BigGAN blocks implemented StudioG AN libary (Kang et al., 2022). AELE-ATTNN is a self-attention block implemented in StudioGAN. BN and LN indicate batch normalization and layer normalization layer, respectively. SPADERESNET
TRANSEN-
CLAYER (dimeoken- diMpiidden headwamb)
is a SPADE block (Park et al. 2019). RESBLOCK-UU UP and -DOwN indicate ResNet blocks implemented in StudioGAN. (p(size) indicates upsampling function PyTorch provides. N is the number of classes in the segmentation mask. m and indicate the batch size and the number of visible instances for given masked image.
15

---

Under review as a conference paper at ICLR 2023
A.3.1 MISSING INSTANCE INFERENCE TRANSFORMER
Missing instance inference transformer consists of 12 layers of transformer encoder layers (Vaswani et al., 2017) with 8 heads. The architectural details are described in Table E We use Adam optimizer (Kingma & Ba 2015) with 31 and 33 of 0.9 and 0.999, respectively. The learning rates linearly increase until 50 epochs and linearly decrease until the end of the training (250 epochs).
A.3.2 INSTANCE SEGMENTATION GENERATOR AND DISCRIMINATOR
We use BigGAN architecture (Brock et al. 2019) implemented in the StudioGAN library (Kang al. 2022). The detailed BigGAN structure can be summarized in Tables and 6 respectively. We use Adam optimizer with 31 and 82 of 0.0 and 0.999, respectively. The learning rates for the generator and discriminator are to 5.0 10 and 2.0 10
A.3.3 BACKGROUND SEGMENTATION COMPLETION NETWORK
For the convolution- background segmentation completion network, iet al. 2015) architecture. For the Transforner-body background segmentation completion network, build layers of the Transformer encoder and add 3 convolution layers and ResNet blocks before and after the Transformer encoder layers. The details of the Transformer-bot body version are shown in Table 7 We optimize the whole segmentation completion network with Adam optimizer with learning rate of 1.0 x 10 and weight decay of 1.0 x 10-4, B1 and B3 for Adam are set to 0.9 and 0.999, respectively.
UNet (Ronneberger
A.3.4 SEGMENTATION COMPLETION NETWORK
We use UNet-like architecture for the segmentation-guiia completion network. We apply SPADERESNETBLOCK to the decoder part to condition the completed segmentation mask into the UNet-like completion network. The architectural details of OASIS (Schönfeld et al. (202I) version of segmentation- -uideed completion network are explained in Tables 8 and 9 We follow the same training details as the original OASIS paper used with a single exception of the objective function. In order to preserve the unmasked region's 's data and generate contents only in the masked region, we apply the OASIS generator and discriminator loss only to the masked region and use additional L2 loss only on the unmasked region. We also add perceptual loss computed on the pre-trained VGG-19 recognition model to encourage fast convergence. For the SPADE (Park et al., (019)) version, we change the OASIS discriminator into the SPADE discriminator. Then, we follow the same training scheme described in the original SPADE paper.
Table 4: Architecture of missing instance inference transformer.
Layer
Input
Output
Operation
Input Layer
(m, k+1, 1)
(m, k+1, 256)
EMBEDDING
Hidden Layer
(m, k+1, 256)
(m, k+1, 256)
TRANSENCLAVERR2566, 2048, 8)
Hidden Layer
(m, k+1,256) (m, k+),
(m, 1+1,,566
TRANSENCLAVER 2048, 8)
Hidden Layer
(m, k+1, 256)
TRANSENCLLYYRR(556 2048,
Hidden Layer
(m, 256)
(m, 1+1,,56
TRANSENCLAYERR 2048, 8)
Hidden Layer
(m, 256)
k+,,
TRANSENCLAYER((256 2048, 8)
Hidden Layer
(m, (4+1,2560
(m, k+1, 256)
TRANSENCLLAVER(2556, 2048, 8)
Hidden Layer
(m, 256)
k+,,
TRANSENCCAAVRR(256 2048, 8)
Hidden Layer
(m, 256)
(m, k1,,+1,250
TRANSENGLAYER(256 2048,
Hidden Layer
k+1, 256)
k+,
TRANSENCLAYER(236 2048, 8)
Hidden Layer
(m, (k+1,256))
(m, 256)
TRANSENGLAYERR(556 2048, 8)
Hidden Layer
256)
k+1,
TRANSENCLAYER(556 2048, 8)
Hidden Layer
(m, k+,, 256)
(m, k+1, 256)
TRANSENCLAYER(256 2048, 8)
Hidden Layer
(m, k+1, 256)
(m, k+1, 256)
TRANSENCLAVERR 2048, 8)
Output Layer
(m, k+1, 256)
(m, k+1, 256)
FC(256, 256), GELU, LN(256)
16

---

Under review as a conference paper at ICLR 2023
Table 5; Architecture for instance segmentation generator.
Layer
Input
Output
Operation
Input Layer
(m,20)
(m,20480)
FC(20, 20480)
Reshape Layer
(m,20480)
(m,4,4,1280)
RESHAPE
Hidden Layer
(m,4, 4, 1280)
(m,8, 8, 640)
(IGGBLooCK(120) 640, True, 20, 128)
Hidden Layer
(m,8, 8, 640)
1m, 16, 320)
BIGGBLOCK 320, True, 20, 128)
Hidden Layer
(m,16, 16, 320)
(m,32, 32, 160)
BIGGBLOCK(320, 160, True, 20, 128)
Hidden Layer
(m,32, 32, 160)
(m,32, 32, 160)
SELF-ATTENTION
Hidden Layer
(m,32, 32, 160)
(m,64, 64, 80)
BIGGBLocK<(600, 80, True, 20, 128)
Hidden Layer
(m,64, 64, 80)
(m,64, 64, 1)
BN, RELU, CONv(80,3, 3,
Output Layer
(m,64, 64, 1)
(m,64, 64, 1)
TANH
Table 6: Architecture for instance segmentation discriminator.
Layer
Input
Output
Operation
Input Layer
(m, 64, 64, 1)
(m, 32, 32, 80)
DBLOCK(3, 80, True)
Hidden Layer
(m, 32, 32, 80)
(m, 32, 32, 80)
SELF-ATTENTION
Hidden Layer
(m, 32, 32, 80)
(m, 16, 16, 160)
DBLOCK(80, 160, True)
Hidden Layer
(m, 16, 16, 160)
(m, 8, 8, 320)
DBLOCK(160, 320, True)
Hidden Layer
(m, 8, 8, 320)
(m, 4, 4, 640)
DBLOCK((300, 640, True)
Hidden Layer
(m, 4, 4, 640)
(m, 4, 1280)
DBLOCK((400, 1280, False)
Hidden Layer
(m, 4, 4, 1280)
(m. 1280)
RELU, GSP FC(1280, 1)
Output Layer
(m, 1280)
(m, 1)
Table 7: Architecture for transformer-body background segmentation completion network.
Layer
Input
Output
Operation
Input Layer
(m, 256, 256, 1)
(m, 256, 256, 64)
EMBEDDING
Hidden Layer
(m, 256, 256, 64)
(m, 128, 128, 128)
CoNv(4, I)
Hidden Layer
(m, 128, 128, 128)
(m, 64, 64, 256)
CoNv(4. 2, 1)
Hidden Layer
(m, 64, 64, 256)
(m, 32, 512)
CoNv(4, 1)
Hidden Layer
(m, 32, 51,
(m, 32,
TRANSENCLAVERR5122 2048, 8)
Hidden Layer
(m, 32, 512)
(m, 32,
TRANSENCLAYYER(512 2048, 8)
Hidden Layer
(m, 32,
32,
TRANSENCLAVER(5122 2048, 8)
Hidden Layer
(m, 32,
(m, 32,
TRANSENCLAYEER(512 2048, 8)
Hidden Layer
(m, 32,
32,
TRANSENGLAYER(5122 2048,
Hidden Layer
(m, 32,
(m, 32,
TRANSENCLAVERR511. 2048, 8)
Hidden Layer
32,
(m, 32, 512)
TRANSENCLAVER(512,, 2048, 8)
Hidden Layer
(m, 32, 512))
(m, 32, 32, 512)
TRANSENCLAYERRE 2048,
Hidden Layer
(m, 32, 512)
(m, 64, 64, 256)
UP(2), ResBlock(3, 1, 1)
Hidden Layer
(m, 64, 64, 256)
(m, 128, 128, 128)
UP(2), ResBlock(3, 1, I)
Hidden Layer
(m, 128, 128, 128)
(m, 256, 256, 64)
UP(2), ResBlock(3, 1, 1)
Oulpul Layer
(m, 256, 256, 64)
(m, 256, 256, 64)
RESHAPE, FC(64, 64), GELU, LN(64)
17

---

Under review as a conference paper at ICLR 2023
Table 8: Architecture for segmentation-guideer image completion generator (OASIS).
Operation
Input
Size
Output
Size
CONCATENATE
Ix
(m, 256, 256, 3)
IMaa
(m, 256, 256, 4)
(m. 256, 256.
1o.V(4.1.1)
JM...
(m. 256,
intero
256. 256.
CoNv(3,2,1)
Sndera
(m, 256, 256, 32)
inter
(m, 128, 128, 64)
CoNV(3,2,1)
intor
(m, 128, 128, 64)
inter
(m. 64, 64, 128)
CONV(3,2.1)
intera
(m. 64, 64, 128)
inters
(m. 32, 32. 256)
CoNV(3.2,1)
inters
(m, 32, 32, 256)
intera
CoNv(3.2,1)
intera
(m, 16, 16, 512)
inters
8. 8, 512)
CoNv(3,2,1)
inters
(m, 8, 8, 512)
infers
(m, 4, 4, 512)
RESHAPE, FC(494=512.256)
inters
(m, 4, 4, 512)
ma
(m, 256)
RESHAPE, FC(494=512.256)
enters
(m, 4,4,
sigma
(m, 256)
NOISE SAMPLING, FC, RESHAPE
(m, 256)
8, 1024)
sigma
(m, 256)
CONCATENATE
zaD
(m, 64, 256, 256)
2p
(m. 64+N+1, 256, 256)
y
256. (m, N. 256, 256)
M
(m, 256, 256)
SPADERESNETBLOCK, CoNv(3, 1,
(m, 8, 8, 1024)
uPo
(m, 8. 8, 512)
(m, 64+N+1, 256, 256)
CONCATENATE
upo
(m, 8, 512)
uPheat
(m. 8, 1024)
inters
(m, 8, 512)
UP(2), SPADERESNETBL.CCK
upocat
(m, 8, 8, 1024)
uP1
16, 16, 512)
(m, 64+N+11 1, 256, 256)
CONCATENATE
(m, 16, 16, 512)
uphoat
(m. 16, 16, 1024)
inters
(m, 16, 16, 512)
UP(2), SPADERESNETBLOCK, CoNv(3.
uPicat
(m, 16, 16, 1024)
uP2
(m, 32, 256)
Ry
(m, 64+N+1. 256, 256)
CONCATENATE
(m, 32, 32, 256)
up2oot
(m. 32, 32, 512)
intery
(m, 32, 32, 256)
UP(2), SPADERESNETB.OCK, CoNv(3,
MP2cot
(m, 32, 32,
NPa
(m, 64, 128)
Fy
(m, 64+N+ 1, 256, 256)
CONCATENATE
ups
(m, 64, 64, 128)
MPSost
(m, 64, 64, 256)
intera
(m, 64, 64, 128)
UP(2), SPADERESNETBH.OCK CoNY(3,
(m, 64, 64, 256)
uPs
(m, 128, 128, 64)
Zy
(m, 64+ 256, 256)
CONCATENATE
up4
(m. 128, 128, 64)
upscat
(m. 128. 128. 128)
intery
(m, 128, 128, 64)
UP(2). SPADERESNETBLOCK, CoNv(3,
uP4cot
(m, 128, 128, 128)
uph
(m. 256, 256, 32)
Fy
(m, 64+N+11 256,
CONCATENATE
ups
(m, 256, 256, 32)
MPPoat
256, 256, 64)
interp
(m, 256, 256, 32)
1,,, TANH
opSent
(m, 256, 256, 64)
Tr
(m, 256, 256, 3)
18

---

Under review as a conference paper at ICLR 2023
Table 9: Architecture for segmentation guided image completion discriminator (OASIS).
Operation
Input
Size
Output
Size
RESBLOCK-DOWN
(m, 256, 256, 4)
down,
(m, 256, 256, 32)
REsBLOCK-DOWN
down1
(m, 128, 128, 128)
down2
(m, 64, 64, 128)
RESBLOCK-DOWN down2
(m, 64, 64, 128)
down3
(m, 64, 64, 128)
RESBLOCK-DOWN down3
(m, 32, 32, 256)
downs
(m, 32, 32, 256)
RESBL.oCK-DoWN
down4
(m, 16, 16, 256)
downs
(m, 16, 16, 512)
RESBLoCK-DoWN
downs
(m, 8, 8122)
downs
(m, 4, 4, 512)
RESBLOCK-UR
down
(m, 4,52))
up1
(m, 8, 8, 512)
CONCATENATE
downs
(m, 8, 8,52)
upicat
(m, 8, 8, 1024)
up1
(m, 8,112)
RESBLOCK-UP -P CONCATENATE
UPicat
(m, (,,8,,10024)
up2
(m, 16, 16, 256) (m, 16, 16, 512)
down4
(m, 16, 256)
uP2cat
up2
(m, 16, 256)
RESBLOCK-UP UP CONCATENATE
uP2cat
(m, 16, 16,512)
ups
(m, 32, 32, 256) (m, 32, 32, 512)
down3
(m, 32,
up3cat
UP3
(m, 32, 256)
RESBLOCK-UP CONCATENATE
UP3cat
(m, 32,
uP4
(m, 64, 64, 128) (m, 64, 64, 256)
down2
(m, 64, 64, 128)
upicat
up4
(m, 64, 64, 128)
RESBLOCK-UP CONCATENATE
up4cot
(m, 64, 64, 256)
ups
(m, 128, 128, 128)
down1
(m, 128, 128, 128)
upscat
(m, 128, 128, 256)
up5
(m, 128, 128, 128)
RESBLOCK-UP 1,,)
uP5cat
(m, 128, 128, 256)
up6 IF
(m, 256, 256, 64)
up6
(m, 256, 256, 64)
(m, 256, 256, N+1)
19

---

Under review as a conference paper at ICLR 2023
A.4
ADDITIONAL QUALITATIVE RESULTS
Input
DETR Seg.
Our Seg.
Target Seg.
Ours
Target
Figure 11: Image completion results.
20

---

Under review as a conference paper at ICLR 2023
Input
DETR Seg,
Our Seg.
Target Seg.
Ours
Target
Figure 12: Image completion results.
21

---

Under review as a conference paper at ICLR 2023
Input
DETR Seg,
Our Seg,
Target Seg.
Ours
Target
Failure case of Refill. From the first the fourth row, Refill generates instances different from the target class. Refill does not aware the order of occlusions as shown in the fifth to the seventh row. Moreover, the eighth and ninth row shows our model suffers from reasoning the right scale and orientation. From the last row, the results show that segmentation mask is also crucial for the performance of generation ability.
Figure 13:
22


Reviews:
[{"review_id": "c8s_6iPeiO", "reviewer": "Reviewer_HUfm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Different from the common inpainting methods, this paper propose a novel image inpainting model tending to fill the missing region with new visual instances instead of filling in the background content. The author first use two transformer-based networks to inpaint the background segmentation map and foreground segmentation map respectively. Then, an unet network is used to hallucinate the missing area with the help of the combined segmentation map. Besides, the author adopts the CLIPScore and VGA to evaluate the context consistency between the original image and the completed image.", "summary_of_the_review": "In summary, this paper proposes a novel image completion method which shows advantage in hallucinating missing region with new visual instances. Substantial experiments prove the effectiveness of the proposed method.", "strength_and_weaknesses": "Strength:\n•\tThe novel image completion pipeline which tends to fill the missing region with new visual instances.\n•\tPrevious context-aware image completion method only focuses on rectangular regions while this method are free to handle masks with arbitrary shapes.\n•\tObtain high quality inpainting results and a number of quantitative results prove its effectiveness.\n\nWeakness:\n•\tThis method utilize a DETR pretrained on COCO which limits the application of this method. For some common used inpainting dataset like CelebA, FFHQ, Places, Paris street View, LSUN, it is hard to acquire the corresponding segmentation map to train the DETR.\n•\tThe author claims ‘COCO-panoptic is more challenging than center-aligned datasets’, but the author should prove the generalization of the proposed method on other domain datasets like face (FFHQ)、buildings (Paris Street View).\n•\tOne important application of inpainting is object removing while this method aims for new object generation. What about the performance of the proposed method on the object removing?\n•\tI find in most figures (Fig 1, Fig 3-5, Fig 8-9), there exist an obvious color discrepancy between the ground truth and the results of this paper. Can you make an explanation?\n•\tWhen processing large missing area, does the segmentation completion network still has the ability to generate new objects or just inpaint with background contents?\n•\tHow is the model complexity and inference time compared to other methods?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "XGiJp3ITFK", "reviewer": "Reviewer_xwWL", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This submission proposes an image completion method to restore the missing semantic instance while preserving the relationship with the original context. To this end, three steps are proposed to complete the inpainting process, including predicting the semantic instance, generating the instance mask, and completing the image. Two evaluation metrics are introduced to assess the completed context.", "summary_of_the_review": "See above", "strength_and_weaknesses": "Strengths\n1)\tTo predict the missing semantic instance and hallucinate it is a challenging but interesting problem.\n2)\tThe writing is easy to follow.\n3)\tThe two introduced evaluation metrics are reasonable and prove the completed context's validity.\n\nWeakness\n1)\tAs I understand, the model performance is highly dependent on the missing instance prediction module. Still, it lacks sufficient explanation and analysis to prove the effectiveness, generalization, and robustness of this module. After all, the semantic instance inference in the segmentation map, especially the corrupted segmentation map is challenging. How about the performance of real-world images?\n2)\tFrom the results, the synthesized content is not consistent in style with the original context, and the completed method needs to be improved.\n3)\tAlthough the proposed method is superior in terms of the two new metrics, it is worse in the metrics of LPIPS and FID. The (novel) application of image completion, which originally intended to complete images to generate realistic content, should be clarified more clearer.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "WKObMjAj2T", "reviewer": "Reviewer_675z", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed a pipeline for image completion, which fills the missing region with a hallucinated instance. It consists of four parts. 1. Object detection via DETR within the masked image. 2. Predict the missing object's category via a multi-head attention network. 3. Generate the foreground & background segmentation mask with GAN. The final segmentation mask is foreground + background. 4. Segmentation guided image completion like SPADE / OASIS.\n\nExperiments are performed on COCO-panoptic and Visual Genome dataset. Four metrics are used, including LIPIPS, FID, CLIPScore, Visual Grounding Accuracy (VGA).", "summary_of_the_review": "I do not recommend acceptance of the paper because of lack of academic novelty. The paper provides an incremental improvement over HVITA, but does not solve critical problems like \"no instance\" or \"instance across missing region\".", "strength_and_weaknesses": "Strength\n1. The proposed pipeline is straightforward and effective. \n2. Improve over HVITA by eliminating the restriction of rectangular missing region.\n\n\nWeakness\n1. Heavily limited to the pretrained detector. The proposed pipeline relies on DETR to provide the objects in the image, subsequent modules is limited to the pre-defined object categories of DETR. In the experiment design, only supported objects are masked as missing. Such heavy dependency limits the real-world applications of the method.\n2. The missing region is within or around the instance bbox, and the method predicts one object regardless of the existence or the number of objects. In many inpainting scenario, the missing region doesn't does not necessarily contain an object in the middle of the region.\n3. Experimental design is biased.\n3.1 The datasets are prepared in favor of the proposed method, i.e., COCO and Visual Genome contains many objects; The missing regions are created around a carefully selected subset of objects.\n3.2 The evaluation metric is biased. The CLIPScore and VGA depends on the object in the missing region. It is obvious that any inpainting method fails as long as it doesn't predict a correct category of object in the missing region.  \n3.3 It will be more convincing to provide methods to determine whether there are objects in the missing (\"no instance\"), and how to deal with cases that half of an object is missing while the other half is not (\"instance across missing region\").\n3.4 Meanwhile, experiments should be conducted on the full (not selected by object category) COCO-panoptic and Visual Genome dataset, with random missing region instead of around object bbox.\n4. It will be interesting to show whether the predicted instance in the impainted image can be detected with DETR. If yes, is the predicted category the same as the output of the \"Missing Instance Inference Transformer\"?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject