Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a Cross-Image Context Memory (CICM) for learning and using the cross-image context to recover the corrupted regions. It tries to provide richer context that benefits the inpainting task. The experimental results demonstrate the effectiveness and generalization of CICM.
strengths_and_weaknesses: Strengths:
1. The idea of using external memory is reasonable. 
2. The paper is clearly written. 

Weaknesses:
1. The method is a little common. 
2. The advantages of external memory need more discussion. 
3. The framework (Figure 2) is a little hard to follow. (c) and (d) provide the details but they are not necessary. 
4. The aim of maximizing/minimizing the inter-/intra-set similarities between the cross-image features is not clear. Why is it useful for impainting? 
questions: 1. Please show some visualization evidence of different/identical set(s).
2. Figure 1 can not show that cross-image feature sets are useful for inpainting. Please provide more clear examples.
limitations: 1. Using external memory can improve the performance, also it will bring a lot of computational costs. Please make some analysis.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes a way to utilize external information for inpainting. To do this, the proposed method maintains a database of clustered features collected from the dataset. Then, for each region's encoded feature, a matching cluster is identified, and the features of that cluster is augmented to the encoded feature in a soft, differentiable way. The proposed method can be an independent add-on to existing inpainting methods, and it consistently improves performance on quantitative metrics that measure similarity with the ground truth, such as PSNR, SSIM, L1, or LPIPS. 
strengths_and_weaknesses: Strength

I find it interesting to utilize external features for better image inpainting. 
The ablations are thoughtful, including single-image vs cross-image context, and different ways to create the bank of features. 
The paper achieves consistently good results than the baseline methods. In particular, it can be plugged into multiple existing methods and improve on all of them. 

Weakness

The exposition is a bit difficult to follow. In general, the paper focuses on how the method is formulated, rather than why. Regarding this, I have a few questions. Please see the Questions section. 

All evaluation metrics measure how much the output matches the ground truth, but it may not directly correlated with realism. For example, since image inpainting is a multi-modal problem with diverse possible outputs, to minimize the L1 loss, the output needs to be at the median of all possible values. In the end, L1 score gives advantage to the outputs that are smooth and low saturation. For example, a pix2pix model trained only on L1 objective (https://phillipi.github.io/pix2pix/images/index_facades2_loss_variations.html) looks unrealistic, even though it does achieve good L1 loss. To separately evaluate realism, metrics like FID are used (ComodGAN, Zhao et al, ICLR2021). 
questions: There is possibility that a baseline method was evaluated incorrectly. It seems that the paper preprocessed the CelebA dataset with facial landmark alignment and cropping (Figure 8 of Supp Mat). However, MISF seems to not have preprocessed the data (their results are on much larger crop around the face, and the faces are not necessarily upright). Therefore, taking the MISF model trained on the raw CelebA data, and evaluating it along with a model trained specifically on aligned images is unfair. Likely because of this, the MISF results of the paper are often blurry (Fig4(a) and Fig8 of Supp Mat), while the results in the original MISF paper not not blurry. It is also wrong to copy the MISF numbers from the original paper (i.e. MISF line with 34.494, 26.635, 21.553, ... in Table 4), and comparing it with the CICM results, because they were in fact run on different datasets (unaligned vs aligned&cropped faces). If MISF were trained and test on the same preprocessing, it may get better results. 

Why is MISF number different between Table 3 and 4? In Table 3, CelebA PSNR is 34.302, 26.387, 21.289, ... In Table 4, it is 34.494, 26.635, 21.553. 

How would compare the quality of this result with ComodGAN (Zhao et al., ICLR2021)? Quantitative comparison is not provided, but ComodGAN seems very nice in qualitative results. 

How is the convolutional encoder (which predicts F_m) trained? If it is trained end-to-end with the losses of Eq9, how are the features C of the cross-image feature sets are updated accordingly? Every update of the network parameter would make the feature Cs outdated. 

I am still not sure why Eq2, 3 and 4 are formulated in the particular form presented. There seems to be connection to online k-means clustering. 

In Eq7, how is the corruption ratio estimated? Is it predicted by the network?
limitations: The authors does address the limitations.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes to use the cross-image context which consists of features learned from different visual patterns to recover the corrupted regions. The proposed method archives state-of-the-art performance on single-image inpainting on multiple datasets.
strengths_and_weaknesses: Pros: 
The proposed CICM achieves stable improvements on all the baseline methods and achieves state-of-art inpainting performance.
The authors conduct extensive experiments for the ablation study to validate the design of context generalization and context augmentation. The design of these modules is validated. 

Cons: 
The core contribution seems not strong enough for me.  As the external memory bank-based method has been adopted in many tasks, what is the unique design to make it suitable for inpainting? On the other hand, the visual quality improvement is also not obvious enough. Even with the proposed CICM, the inpainting results are not as attractive as LaMa and RePaint.
 
The writing needs to be improved. There are many sentences that are not meaningful but very complicated (e.g., lines 36 to 39). 

How is the external memory bank-based method compared to gan prior-based methods such as LAMA?
Is it possible to visualize the features learned in CICM? It is interesting to see how the CICM is learned during the training. 
 
Suggestions:
Missing reference in Figure 4. 

questions: Please address my concerns shown in the weaknesses.
limitations: N/A
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes an image inpainting algorithm that learns visual context features across different images and saves them in an external memory (CICM). These features are used to augment regional features of the corrupted input image, which may result in better completion quality than relying on features inside the single image. The proposed approach outperforms existing models on the public datasets.
strengths_and_weaknesses: * Strengths:
  - The proposed approach based on cross-image context memory is novel. It saves higher level richer visual context thus is unlike previous memory based methods such as TMAD [42] and SRM [43].
  - The proposed approach is highly effective, outperforming recent existing works (Tables 4 and 5).
  - The proposed approach is flexible and general, capable of extending several existing frameworks with consistent enhancements (Table 3).
  - Ablation studies on internal component of the proposed approach have been thoroughly made thus proved their importance (Tables 1 and 2).
  - Several extensions and variants of the proposed approach have been explored in the supplementary materials, all showing meaningful results.

* Weaknesses:
  - It is unclear how the CICM is initialized at the beginning of the training.
  - It is unclear which architecture is used as their default backbone network.
  - The use of an external memory bank increases the ram usage, parameters and FLOPs quite a bit.
  - Depending on the scale of CICM and device property, the searching processes may become a bottleneck of the inference speed.


----- Comments after reading the rebuttal -----

I believe that the proposed work describes a very interesting algorithm based on CICM, yet unseen from existing works. As addressed by the authors, the CICM is fast to search into and effective to produce higher quality generation by utilizing compact cross-image information unavailable within the input image. The quality gains are quite clear without much sacrificing the efficiency. My final recommendation is still to strongly support to accept the paper.
questions: How is the CICM initialized?

What is the backbone network of the proposed model in Tables 4 and 5?

Would the quality enhancement by applying CICM scheme be outstanding enough compared to the complexity increase (i.e., trade-off)?

The search into the CICM is not a bottleneck of the model performance, which is not addressed?
limitations: Yes, they addressed the limitations in Sections 3 and 4 of the supplementary materials, which include failure cases, memory increase, cross model and cross dataset scenarios, and societal impacts. These discussions seem to have been thoroughly made.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes the Cross-Image Context Memory (CICM) for learning and using the cross-image context to recover corrupted images. CICM consists of multiple sets of cross-image features learned from the image regions with different visual patterns. The regional features are learned across different images, thus providing richer context that benefits the inpainting task. The experimental results demonstrate the effectiveness and generalization of CICM, which achieves state-of-the-art performances on various datasets for single image inpainting.
strengths_and_weaknesses: Strengths 
1）	The utilization of the cross-image context to assist in image inpainting is reasonable and the proposed cross-image context memory is somewhat novel and can also be generalized to the existing inpainting models.
2）	The experiments are sufficient, and the internal study is nice to show the effectiveness of the CICM
3）	The presentation is clear, and the reference is also adequate.

Weaknesses
1)	Besides the number of feature sets and the size of each set, the resolution of the regional features and the number of layers adopting the CICM are also vital settings that affect the image inpainting performance, but there seems no explanation and analysis for them.

Minor:
1)	In line 74, symbols of H and W are used to denote the height and width of the image and the feature, but they are actually different.
2)	In line 117, what is the value of the momentum factor.
3)	In line 168, “four groups” is “three groups”.

questions: 1)	Five losses are used to train the model. Is there any balance for their contributions? Since there are different terms in each loss, why are the magnitude of these loss values the same as in Fig. 5 in the supplementary?
2)	How are the anchor features and feature sets initialized?

limitations: N/a
review_title: null

Meta-review: metareview: The paper discusses how to use external information for inpainting. Reviewers appreciated the idea but raised concerns regarding limited novelty, use of the proposed method for inpainting, baselines being evaluated incorrectly, and missing ablations. The rebuttal was able to address most of the concerns and reviewers remained positive. AC concurs and doesn't find reasons to overturn an unanimous majority recommendation.
confidence: Certain
Decision: Accept