Overview of Advanced Convolution layers | by Jehill Parikh | Towards Data Science

Convolution layers are fundamental building blocks of computer vision architectures. Neural networks employing convolutions layers are employed in wide-ranging applications in Segmentation, Reconstruction, Scene Understanding, Synthesis, Object detection and more
The goal of this post is to provide a summary and overview of advanced convolution layers and techniques which have emerged in the recent literature. We start with basics of convolution, for completeness, however, more rigorous explanations can be obtained from references within.
Convolution: Mathematically speak convolution is an “operation” performed to combine two signals into one, below is an illustration from wikipedia which highlights convolution between two functions/signals f(t) and f(t-z). Convolution to obtain the (f*g)(t)
The main convolution operations in deep learning are
“2D” Convolution:
“2D” Convolution:
Pictorially we convolve “slide” a kernel (green size) over an image (blue) and learn weights for these kernels. This kernel’s spatial extent (F) is 3 and filters i.e. depth of the kernel is 1, therefore number of weights are 3*3*1=9. We can skip pixels by “stride” and pad regions our original image, here the stride is 0
Convolution block of accepts a image or feature map of size W1×H1×D1 and kernel of size (F*F*K) and typical requires four hyper-parameters:
Number of filters K: K
Their spatial extent F: F
The stride S
The amount of zero padding: P : if we zero pad any image
Number of parameter are (channels*F*F)*K
Typically shape of the weight matrix is (F, F, Channels, K)
These there operations combined provide a final output feature map of W2*H2*D2 for details of working see post
W2 =(W1−F+2P)/S+1W2=(W1−F+2P)/S+1
H2 = (H1−F+2P)/S+1H2=(H1−F+2P)/S+1
With convolution, two additional operations are employed
Max-pool: this operation reduces the number dimension of the images: generally a 2*2 filter in “max” pool operation is employed. The filter replaces each 2*2 block in the image/feature map with the max of that block, this is reduced the size of the feature maps for the following layer.
Non-linearity (relu, tanh etc) are employed prior to max-pool operation to add non linearity to the operation [TODO: future work: add notes on performance of the various non-linearity]
Spatial extent of filter (F) is the main contributor of the number of weights for each kernel. As explained in CS231n lecture notes with F=3 and F=7 there is a three fold increase in number of weights. Typically, a full deep-net consists of multiples layers of CONV + RELU + POOL operations to extract features. Thus generally F=3 is employed as a trade off to learn features vectors at each layer without increasing cost of computational cost.
Please see the computation considerations section on C231n lecture notes as additional reference
Key architectures for object classification tasks which use these three layers are well summarised in the CS231n notes, they are LeNet, AlexNet, ZFNet, GoogLeNet, VGGnet, Resnet. These were developed towards winning the Imagenet challenge. Over the years, there was a trend for larger/deeper the network deeper to improve performance.
Residual/skipped connection
These were introduced in 2015 by a Microsoft team to maintain accuracy in deeper networks as part of the ImageNet challenge. A single network skipped connection is shown below, it aims to learn Residual R(x). In deeper networks this allows to learning “residual information” at each layer. Hence the name residual connection. This technique has experimentally has proven to increase accuracy in deeper networks. Implementing a residual/skipped connection is very straight forward, as shown below. Skipping the connection also allows us to over come to issue of vanishing gradients, in deep layers, and speeds up training. This experimental results has been widely adopted across a range of computer vision algorithms since original introduction. Variants of the resnets are highlighted in this post, and additional detail and illustrations please see the blog.
R(x) = Output — Input = H(x) — xdef res_net_block(input_data, filters, conv_size):  x = layers.Conv2D(filters, conv_size, activation='relu',      padding='same')(input_data)  x = layers.BatchNormalization()(x)  x = layers.Conv2D(filters, conv_size, activation=None,         padding='same')(x)  x = layers.BatchNormalization()(x)  x = layers.Add()([x, input_data])  x = layers.Activation('relu')(x)  return x
Above layers are critical components of neural networks employed for computer vision task, and find application in wide range of domains and different architecture. Now we turn to more specialised layers.
Convolution transpose/strided convolutions
Convolution operation with stride (≥2) and/or padding reduces the dimension of the resultant feature map. Convolution transpose is the reverse process employed to learn kernels to, up-sample features maps to larger dimensions. With stride >1 being typically employed to upsample the image, this process is well illustrated in a post by Thom Lane and below, where are 2*2 input with padding and strided convolution with a 3*3 kernel leads to a 5*5 feature map
Strided convolutions find wide ranging application in different areas
U-nets (medical image segmentation)Generative models: GAN: Generators VAE: Decoder, up-sampling layers
U-nets (medical image segmentation)
Generative models: GAN: Generators VAE: Decoder, up-sampling layers
Implementation: All major DL frameworks have strided convolution, then need to employed with proper initialisation i.e. random or Xavier initialisation.
Masked and gated convolution
Masked and Gated convolution started gaining popularity around 2016, in there seminal work Aaron van den Oord, et al introduced Pixel RNN and Pixel-CNN. These are autoregressive approaches to sample pixel from a probability distribution conditional on the previous pixels.
Since each pixel is generated via conditioning on previous pixels, to ensure conditioning on pixel from the left and top rows mask are employed while applying convolution operations. Two types of masks are Mask: A, used in first channel and prior pixels. Mask: B, mask B all channels and prior pixels, following layers, both available here
Notes: Inputs must be binarized: 28 e.g. 256 bits of colour, or each sample is between 0–256 in a RGB image
Masked gated convolutions were introduced to avoid blind-spots in masked convolutions. Ord et al, 2016 proposed to isolate horizontal and vertical stacks i.e gates with 1*1 convolution, with residual connections in the horizontal stacks, as shown below. Residual connections in vertical stacks didn’t offer additional performance improvement, therefore were not implemented.
Implementation of masked gate convolutions is available here.
Applications: Mainly employed in decoders for e.g. in VAE frameworks for prior sampling to avoid issues with training GAN’s such as mode collapse and generate high resolution images.
Pixel CNN decoder: Pixel conditioned on previous values, leading better performance. Longer inference/sampling as needs to be done on pixel by pixel basis. These were optimised in PixelCNN++ and implementation available on available on Tensorflow Probability.Pixel-VAE: combines, each dimension is diagonal element of the covariance matrix and this Gaussian assumption leads to poor sample images, so combines a traditional decoder with PixelCNN to help “small scale/similar” aspects of the distribution and this puts less demands on latent to learn more global information, demonstrated via improvements to KL term, leading to improved performance. Implementation see hereVQ-VAE and VQ-VAE2: uses Pixel-CNN in latent code map of VAE to avoid Gaussian assumption of the latent variables all together, leading to diverse images in higher dimensions. VQ-VAE implementation were open sourced by the authors, and other implementation are also widely available
Pixel CNN decoder: Pixel conditioned on previous values, leading better performance. Longer inference/sampling as needs to be done on pixel by pixel basis. These were optimised in PixelCNN++ and implementation available on available on Tensorflow Probability.
Pixel-VAE: combines, each dimension is diagonal element of the covariance matrix and this Gaussian assumption leads to poor sample images, so combines a traditional decoder with PixelCNN to help “small scale/similar” aspects of the distribution and this puts less demands on latent to learn more global information, demonstrated via improvements to KL term, leading to improved performance. Implementation see here
VQ-VAE and VQ-VAE2: uses Pixel-CNN in latent code map of VAE to avoid Gaussian assumption of the latent variables all together, leading to diverse images in higher dimensions. VQ-VAE implementation were open sourced by the authors, and other implementation are also widely available
Invertible convolutions
Invertible convolutions, are based on normalising flows, are currently applied in generative models to learn underlying probability distribution p(x) of image in computer vision. The maintain motivation is to provide a better loss function i.e. negative log likelihood.
The two most common generative frameworks modelling suffer from approximation inference time issues, i.e. loss function function for VAE (evidence based lower bound i.e. ELBO) is an approximation is lower bound on log-likelihood, therefore inference/reconstruction is approximation. Adversarial loss employed in GAN’s is a “search” based approach and suffer issues with sampling diversity and are hard to train i.e. mode collapse.
Mathematical preliminaries of Normalising flows are best outlined in the Stanford class notes here, we use there summary.
In normalising flow, the mapping between random variable Z and X, given by, function f, paramaterized θ, which is deterministic and invertible such that
Then probability distributions of X and Z can then be obtained using the change of variable formulation, in this case the two probability distributions are related via determinant term. Loosely speaking the determinant is the “scaling” or normalising constant.
Here,
“Normalising” means that the change of variables gives a normalised density after applying an invertible transformation“Flow” means that the invertible transformations can be composed with each other to create more complex invertible transformations.
“Normalising” means that the change of variables gives a normalised density after applying an invertible transformation
“Flow” means that the invertible transformations can be composed with each other to create more complex invertible transformations.
Note: function can be a single function, or a series of sequential function, transforming generally, transforming from simple e.g. latent vectors (Z) to complex distributions e.g. (X) images. This formulation allows to us “exactly” between transform between two distribution, and thus can derive the negative log likehood loss function, see the lecture for derivation.
Normalising flow and neural networks: Works of Dinh et al, 2014 (NICE) and Dinh et al, 2017 (Real-NVP), started providing neural network architecture, to employ normalising flow for the density estimation. Glow from Kingma et al, is the current (2018) state of art, which builds on these works, In which which introduced 1*1 invertible convolution, to synthesis high resolution images.
The key novelty was to reduce the computation cost of the determinant term for the weight matrix, for 1*1 learning invertible convolutions. This was achieved with LU decomposition with permutation, i.e. PLU decomposition, for the weight matrix. Random permutations were employed to maintain “flow” at each iteration. The mathematical details are covered in section 3.2, of the paper they also provide an implementation using numpy and tensorflow, for easier interrogation.
These were further generalised to N*N convolutions by Hoogeboom, et al, please see the blog post for additional details and implementations (with optimisation). Our aim was just to highlight these models, for more comprehensive details we refer the reader to, CS236 lectures 7 and 8 and Glow paper and blog post by Lilian Weng.
Application area: Image synthesis and generative modelling