Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposed a pipeline for image completion, which fills the missing region with a hallucinated instance. It consists of four parts. 1. Object detection via DETR within the masked image. 2. Predict the missing object's category via a multi-head attention network. 3. Generate the foreground & background segmentation mask with GAN. The final segmentation mask is foreground + background. 4. Segmentation guided image completion like SPADE / OASIS.

Experiments are performed on COCO-panoptic and Visual Genome dataset. Four metrics are used, including LIPIPS, FID, CLIPScore, Visual Grounding Accuracy (VGA).
strength_and_weaknesses: Strength
1. The proposed pipeline is straightforward and effective. 
2. Improve over HVITA by eliminating the restriction of rectangular missing region.


Weakness
1. Heavily limited to the pretrained detector. The proposed pipeline relies on DETR to provide the objects in the image, subsequent modules is limited to the pre-defined object categories of DETR. In the experiment design, only supported objects are masked as missing. Such heavy dependency limits the real-world applications of the method.
2. The missing region is within or around the instance bbox, and the method predicts one object regardless of the existence or the number of objects. In many inpainting scenario, the missing region doesn't does not necessarily contain an object in the middle of the region.
3. Experimental design is biased.
3.1 The datasets are prepared in favor of the proposed method, i.e., COCO and Visual Genome contains many objects; The missing regions are created around a carefully selected subset of objects.
3.2 The evaluation metric is biased. The CLIPScore and VGA depends on the object in the missing region. It is obvious that any inpainting method fails as long as it doesn't predict a correct category of object in the missing region.  
3.3 It will be more convincing to provide methods to determine whether there are objects in the missing ("no instance"), and how to deal with cases that half of an object is missing while the other half is not ("instance across missing region").
3.4 Meanwhile, experiments should be conducted on the full (not selected by object category) COCO-panoptic and Visual Genome dataset, with random missing region instead of around object bbox.
4. It will be interesting to show whether the predicted instance in the impainted image can be detected with DETR. If yes, is the predicted category the same as the output of the "Missing Instance Inference Transformer"?
clarity,_quality,_novelty_and_reproducibility: Lack of academic novelty. In terms of weakness 3, the authors may argue that predicting objects is the major contribution of the paper. However, that also means predicting a limited set of objects in limited scenarios is the only thing that the proposed method can do. Indeed it is an improvement over HVITA, but it is more like an incremental one, that follows the path of HVITA and equipped with several off-the-shelf components.
summary_of_the_review: I do not recommend acceptance of the paper because of lack of academic novelty. The paper provides an incremental improvement over HVITA, but does not solve critical problems like "no instance" or "instance across missing region".
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This submission proposes an image completion method to restore the missing semantic instance while preserving the relationship with the original context. To this end, three steps are proposed to complete the inpainting process, including predicting the semantic instance, generating the instance mask, and completing the image. Two evaluation metrics are introduced to assess the completed context.
strength_and_weaknesses: Strengths
1)	To predict the missing semantic instance and hallucinate it is a challenging but interesting problem.
2)	The writing is easy to follow.
3)	The two introduced evaluation metrics are reasonable and prove the completed context's validity.

Weakness
1)	As I understand, the model performance is highly dependent on the missing instance prediction module. Still, it lacks sufficient explanation and analysis to prove the effectiveness, generalization, and robustness of this module. After all, the semantic instance inference in the segmentation map, especially the corrupted segmentation map is challenging. How about the performance of real-world images?
2)	From the results, the synthesized content is not consistent in style with the original context, and the completed method needs to be improved.
3)	Although the proposed method is superior in terms of the two new metrics, it is worse in the metrics of LPIPS and FID. The (novel) application of image completion, which originally intended to complete images to generate realistic content, should be clarified more clearer.

clarity,_quality,_novelty_and_reproducibility: The writing is easy to follow and the task is interesting, but the technical contribution and experiments still need to improve.
summary_of_the_review: See above
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Different from the common inpainting methods, this paper propose a novel image inpainting model tending to fill the missing region with new visual instances instead of filling in the background content. The author first use two transformer-based networks to inpaint the background segmentation map and foreground segmentation map respectively. Then, an unet network is used to hallucinate the missing area with the help of the combined segmentation map. Besides, the author adopts the CLIPScore and VGA to evaluate the context consistency between the original image and the completed image.
strength_and_weaknesses: Strength:
•	The novel image completion pipeline which tends to fill the missing region with new visual instances.
•	Previous context-aware image completion method only focuses on rectangular regions while this method are free to handle masks with arbitrary shapes.
•	Obtain high quality inpainting results and a number of quantitative results prove its effectiveness.

Weakness:
•	This method utilize a DETR pretrained on COCO which limits the application of this method. For some common used inpainting dataset like CelebA, FFHQ, Places, Paris street View, LSUN, it is hard to acquire the corresponding segmentation map to train the DETR.
•	The author claims ‘COCO-panoptic is more challenging than center-aligned datasets’, but the author should prove the generalization of the proposed method on other domain datasets like face (FFHQ)、buildings (Paris Street View).
•	One important application of inpainting is object removing while this method aims for new object generation. What about the performance of the proposed method on the object removing?
•	I find in most figures (Fig 1, Fig 3-5, Fig 8-9), there exist an obvious color discrepancy between the ground truth and the results of this paper. Can you make an explanation?
•	When processing large missing area, does the segmentation completion network still has the ability to generate new objects or just inpaint with background contents?
•	How is the model complexity and inference time compared to other methods?


clarity,_quality,_novelty_and_reproducibility: The paper is well organized and clearly written. The paper appears to be technically sound and maybe it is a little difficulty to reproduce the results.
summary_of_the_review: In summary, this paper proposes a novel image completion method which shows advantage in hallucinating missing region with new visual instances. Substantial experiments prove the effectiveness of the proposed method.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper receives 2x marginally below the acceptance threshold and 1x marginally above the acceptance threshold. The overall rating is leaning towards rejection. The major weakness of the paper is the lack of academic novelty where paper provides an incremental improvement over HVITA, but does not solve critical problems like "no instance" or "instance across missing region". The model performance is highly dependent on the missing instance prediction module. Nonetheless, there is a lack of sufficient explanation and analysis to prove the effectiveness, generalization, and robustness of this module.

The metareviewer carefully reads the authors' response to the lack of novelty over HVITA and finds that besides introducing new evaluation metrics and claiming the Refill model outperforms HVITA, there is no detailed explanation of what are the technical differences between HVITA and the proposed model. Although there is a 1x marginally above the acceptance threshold, the strengths highlighted by the reviewer do not address the novelty claim of the proposed method over HVITA. 
justification_for_why_not_higher_score: This paper receives 2x marginally below the acceptance threshold and 1x marginally above the acceptance threshold.
justification_for_why_not_lower_score: NA
Decision: Reject