“Lines First, Color Next” An Inspirational Deep Image Inpainting Approach
By: Chu-Tak Li
Tags: Deep Learning, Convolutional Network, Image Inpainting, Image Processing, Generative Adversarial

Perceptual Loss

Here, I briefly mention the perceptual loss to show its similarity to the feature matching loss. Φ_i(I) represents the feature maps at the i-th layer of Φ given the input I. In this work, the authors use relu1_1, relu2_1, relu3_1, relu4_1, and relu5_1 of a pre-trained VGG-19 network to calculate the perceptual loss and the style loss. For readers who want to know more about the perceptual loss and the style loss, you may refer to my last post.

Overall Loss Function

This is the overall loss function to train the image completion network (i.e. the second generator). λ_l1, λ_adv,2, λ_p and λ_s are the weights to balance these four loss terms. In their experiments, these 4 weights are set to 1, 0.1, 0.1, and 250 respectively. Different from the first edge prediction network, SN is only applied to the discriminator for this image completion network to speed up the training process.

Experiments

Edge Information and Masks

For training the edge generator G_1, the authors employ a very famous conventional edge detection algorithm named Canny edge detector. I will not go through the Canny edge detector here, interested readers may google it for more details. Anyway, for Canny edge detector, we have to input a parameter σ which would affect the quality of the detected edge image. The authors found that σ=2 gives the best edge image for the task of image inpainting. I will talk about this ablation study later on.

For the types of the masks, two types of masks are used in this paper. The first type is a regular rectangular missing region with fixed size (25% of the pixels are masked) at a random location in an image. The second type is the irregular mask introduced in the work of Partial Convolution we have introduced last time.

The authors evaluated their proposed approach on CelebA, Places2, and Paris StreetView datasets. As you can see, these are some commonly used datasets for evaluating deep image inpainting approaches.

Qualitative Comparison

Figure 3. Some inpainting results by using the proposed EdgeConnect. From left to right. Ground truth images, Input masked images, predicted edge maps, and completed images. Image by Kamyar Nazeri et al. from their paper [1]

Figure 4. Visual comparison of different state-of-the-art approaches. Image by Kamyar Nazeri et al. from their paper [1]

Figure 3 shows some inpainting results using the proposed method, EdgeConnect. Please zoom in for a better view of the quality of the completed images. Figure 4 shows the visual comparison of different inpainting approaches. Note that Figure 4(c) and (d) represent the Contextual Attention and the GLCIC we have covered before. For Ours (Canny), the authors try to provide the ground truth edge information (not available in practice) to the image generator, hence the generator only needs to fill in the local details with full edge information. Again, please zoom in for a better view. You can see that the proposed method, Figure 4(e), offers the best inpainting results. If the ground truth edge information is available, Figure 4(f), even better inpainting results can be produced.

Quantitative Comparison

Table 1. Quantitative comparison of different state-of-the-art approaches on the Places2 dataset. Data by Kamyar Nazeri et al. from their paper [1]

Table 1 lists out some commonly used objective numerical metrics for the task of image inpainting. Note that for SSIM and PSNR, the higher the better; while for l1(%) and FID, the smaller the better. FID can be regarded as a metric to estimate the visual quality. The last column (Canny) means that the full edge information is available for the second image generator. We can see that in most of the situations, the proposed EdgeConnect (i.e. edge prediction first then image completion) offers the best numerical metrics.

Figure 5. The effect of various mask sizes on PSNR and FID on the Places2 dataset. Image by Kamyar Nazeri et al. from their paper [1]

Figure 5 shows the effect of various mask sizes on PSNR (right) and FID (left) on the Places2 dataset. As you can see, when the mask ratio becomes larger, the task becomes much more difficult.

The authors also did visual turing tests to evaluate their inpainting results. If you are interested on it, please refer to their paper for details.

Ablation Study

Table 2. Quantitative comparison of inpainting results with the use of the edge information (Yes) and without the use of the edge information (No). Data by Kamyar Nazeri et al. from their paper [1]

First of all, the authors provide evidence that the use of edge information is useful for the task of image inpainting. Table 2 lists out the related numerical metrics and shows that the use of the edge information is helpful to enhance the inpainting results.

Figure 6. The effect of different values of the parameter σ on PSNR and FID on three different datasets. Image by Kamyar Nazeri et al. from their paper [1]

Then, they have tried different values of the parameter σ to obtain different edge images and see the effect of using different edge images on the final inpainting results. Figure 6 shows the effect of different σ on PSNR and FID on three different datasets. As you can observe, σ=2 (or 2.xx) offers the best final inpainting results (best PSNR and FID).

Figure 7. The effect of different parameter σ on the inpainting results. Image by Kamyar Nazeri et al. from their paper [1]

Figure 7 shows the effect of different σ on the edge images and the inpainting results. Simply speaking, as you can see, when σ is too small, we will have too many small edges and this may be too complicated for image completion. On the other hand, when σ is too large, it is difficult to detect edges and hence the edge maps may not be useful for image completion. Therefore, suitable σ is crucial for producing good inpainting results.

Apart from the Canny edge detector, the authors also tried to employ better edge detection algorithm to see any further improvements can be brought. They found that there is no any obvious improvements, hence they chose to use the Canny edge detector. Of course, the choice of the edge detection algorithm has not been fully explored. Again, for readers who want to know more about this experiment, please refer to the paper.

Limitations and Applications

Figure 8. Some inpainting results where the edge generator fails to generate satisfactory edge information. Image by Kamyar Nazeri et al. from their paper [1]

Figure 8 shows some failure cases in which the edge generator cannot produce satisfactory edge information. As you can see, when the missing region is too large to be filled and the scene is too complex, it is still difficult to complete the image with good visual quality.

Figure 9. Possible extension to an interactive image editing tool. (a) Image A (b) Image B (c) Edge map (black: the edge image of the left-half of (a) red: the edge image of the right-half of (b)) (d) Inpainting results (input image is (a) with the right-half is masked and (c) is the edge information used for completion). Image by Kamyar Nazeri et al. from their paper [1]

Figure 9 shows a possible extension of the proposed EdgeConnect to an interactive image editing tool. You can see that it is possible to combine the characteristics of two images in the form of image inpainting. Please zoom in for a better view of the results.

Figure 10. Some examples of object removal and image editing using the proposed EdgeConnect method. Left: original image. Middle: mask images with unwanted objects and optional edge guidance. Right: Inpainting results. Image by Kamyar Nazeri et al. from their paper [1]

Figure 10 shows some examples in which unwanted object is removed from the original images. Note that edge guidance is optional and you may zoom in for a better view of the results.

Conclusion

Clearly, the main idea of this paper is to separate the task of image inpainting into two simpler steps, namely edge prediction and image completion based on the predicted edge map. They first predict the edges in the missing regions, then complete the image based on the predicted edge information. By doing this, sharper inpainting results are obtained instead of blurry results. In fact, most of the techniques used in this paper have been introduced in my previous posts. If you have read my previous posts, you may find that it is easy to understand the idea of this paper. For readers who are interested in the idea of this paper, I highly recommend you to read their paper for further details such as the detailed network architecture and the training procedure.

Takeaways

Actually, we have gone through most of the commonly used technqiues for deep image inpainting. Now, we can handle different types of masks such as regular rectangular masks and irregular masks. Image inpainting has various possible applications such as unwanted object removal and other image editing tasks (combine two images half-half as shown in Figure 9). However, same as before, it is still very difficult to complete images with complicated structures and large mask ratio. Anyway, we still have a long way to go for deep image inpainting. Let’s learn and read more related papers together! :)

What’s Next?

Next time, we will dive into another very popular deep image inpainting paper which can be regarded as an enhanced version of Contextual Attention (DeepFillv1), Partial Convolution, and EdgeConnect. Looking forward to learning together! :)

References

[1] Kamyar Nazeri, Eric Ng, Tony Joseph, Faisal Z. Qureshi, Mehran Ebrahimi, “EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning,” Proc. International Conference on Computer Vision (ICCV), 2019.

[2] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida, “Spectral Normalization for Generative Adversarial Networks,” Proc. International Conference on Learning Representations (ICLR), 2018.

Thanks again for reading my post. If you have any questions, please feel free to send my an email or leave comments here. I am happy to hear from you and any suggestions are welcome. Hope to see you next time! :)