CNN Basics — Part 2. In Part1, I have covered the mechanics…
By: Ozgur Guler
Tags: Machine Learning, Convolutional Network, Image Processing

In Part1, I have covered the mechanics of the convolution operation and how it helps with standard image processing tasks e.g. feature detection.

In Part2, I will be covering new types NN layers used for building CNN’s, and techniques to decrease computational load required for a CNN.

In a typical CNN we see three types of layers

Convolutional Layer (CONV)

Pooling layer (POOL)

Fully connected layer (FC)

POOL layers are mainly to reduce the size of representation to speed up computation as well as to make the features the CNN detects more robust. In an input nxn matrice, pooling operation carves submatrices of mxm out of the input matrix and applies a specified operation on the submatrices. e.g. for max pooling, the matrix cell with the maximum value is chosen and mapped to the output, or with average pooling we take the average of cells within the filter. That way a feature map is created that confirms the presence of certain features in the input image. Formulas in standard convolution operation for output dimensions still holds (n+2p -f )/s+1. Padding in general is not used with pooling operations. Typical filter sizes are n=2, s = 2 which reduces the output dimensions to half of the input matrix. In a CNN, typically nH and nW (height and width of the output matrix) of the input image decreases and the number of channels (nC) increases.

In a typical NN, the standard forward propagation for each module includes a linear operation z = w .a + b. Output is processed by a non-linear activation function (e.g. sigmoid, ReLU). The output is than fed as an input to the next layer. Operation can be depicted as below where z[l+1] is the output of each layer l.

z[l+1]=w[l+1] a[l] + b [l+1] , a[l+1] = g(z[l+1]) activation function

z[l+2]=w[l+2] a[l+1] + b [l+2], a[l+2] = g(z[l+2])

In RESNET’s, a[l] from layer l, is fed to a layer is deeper in the NN architecture. This makes a[l+2] = g(z[l+2] + a[l]), since a[l] is fed into l+2 layer. Because of NN regularisation, a[l+2] becomes close to a[l] which diminishes the affect of the added 2 convolution layers in the output. This means RNN’s can easily learn the identity function. If the additional 2 CNN blocks learn any useful feature it is reflected in the model output, otherwise the additional layers do not significantly detoriate the training performance as seen in very deep NN’s where training performance diverges after a certain depth of the NN.

1X1 convolutions are generally used to reduce the number of channels in the input.

If you would not want to commit to a single type of filter, you can use filters of different sizes and stack the outputs. This is called an “inception” network. 1X1 filters are quiet useful to reduce the reduce the increased processing requirements of an “inception” network since the convolution operation is repeated with many seperate filters of differing length. To reduce input dimentions a “bottleneck” layer of 1X1X [reduced amount of channels] is used for convolution, a second convolution will be required to reach the final output.

With pooling, Resnets, 1X1 convolution and Inception networks we practically decrease the compute requirements of running the CNN.

Next I will cover detection algorithms.