Inpainting of Irregular holes using Partial Convolution: Paper summary
By: Vaibhav Tiwari
Tags: Inpainting, Convolutional Network, Unet, Autoencoder, Irregular Holes

In my previous blog, I’ve talked about the basics of Inpainting and how advancements in the methods show the promising future of Inpainting. The above-mentioned blog ended with the Introduction of Partial Convolutions based Image inpainting and how it has sorted out the various problems that were faced by the past models. Here, I will discuss the crux of ‘Image Inpainting for Irregular Holes Using Partial Convolutions’ paper in detail.

What is Partial Convolution?

Partial Convolution performs the normalization of the output to adjust for the fraction of missing data. A partial convolution layer comprises masked and re-normalized convolution operation followed by a mask-update setup. It does not consider the patch pixels of the image while convolution.

Partial Convolution is defined as:

Partial Convolution, Courtesy: Paper

W= Kernel weight

X= Feature values for the current sliding window

M= Binary mask corresponding to X

(.)= Element wise multiplication

1= Matrix with all elements as ‘1’ and dimension equal to that of ‘M’

b= bias

An example would give a better understanding. Consider a small part X of a big image:

Xp0= X surrounded with hole region completely

1p0= Analogous to ‘M’ with value ‘1’ for defined and ‘0’ for the patched region

1p1= Analogous to 1 defined previously

Partial Convolution example, Courtesy: Paper

x’ = Wt( Xp0 . 1p0) sum(1p1)/sum(1p0) + b

In the first convolution, only non-patch pixels are considered and the product of patch pixel with convolution kernel weight is set to ‘0’.

Following the red box in the above example, Xp0 is element-wise multiplied with 1p0 and normalized with the help of 1p1.

is element-wise multiplied with and normalized with the help of The output sum is upscaled according to the number of missing pixels with the use of term Sum(1)/Sum(M).

For convolutions after the very first layer, the same thing is followed except the definition of a non-patch feature is updated to all features that have at least one non-patch input pixel in their receptive field.

The receptive field is the number of input pixels that impact the convoluted output pixels.

Network Architecture

UNet-like architecture is used in the original paper for performing the ‘Completion network’ operation (‘Completion network’ explained here). Instead of using Normal CNNs, Partial Convolution layers are used accompanied with skip links. As the architecture is long, minute details are further required in the later layers in order to produce better results. This information is transferred by concatenating Image and Mask from the nth encoder layer with the equivalent decoder layer, this is the function of a Skip link.

Courtesy : Paper

A UNet example from this paper

ReLU is used in the encoding stage and LeakyReLU with alpha = 0.2 is used between all decoding layers. The encoder comprises eight partial convolutional layers with stride=2. The kernel sizes are 7, 5, 5, 3, 3, 3, 3 and 3. The number of channels are 64, 128, 256, 512, 512, 512, 512, and 512.

(I won’t dive into the Losses calculation as it is completely mathematical and quantitative)

Bonus: Partial Convolutions for Padding

Traditionally accepted Padding techniques like Zero padding, Reflection padding, and Replication padding attempt to pad by using the data values that are available near to the border of the image. These techniques result in unrealistic image patterns as only a part of the features are used. To solve these issues, Partial convolution can be used for padding while assuming the outermost padded boundary of the image as a Patch and filling it with weighted data values.