Beginners Guide to Convolutional Neural Network from Scratch — Kuzushiji-MNIST
By: Satsawat Natakarnkitkul
Tags: Convolutional Network, Keras, Image Classification, Machine Learning, Data Science

Convolutional Neural Network Layers

Figure 6: Example of ConvNet architecture

There are several main layers implemented in ConvNets.

Conv2D layer

It is a 2-Dimension convolutional layer, mainly it is used as the layer to extract features from the raw input image. The first layer is responsible for capturing low-level features such as edges, color, gradient orientation. With added layers, the architecture will attempt to capture high-level features.

Figure 7: 1-d convolution

Conv2D filters extend through the three channels in the image (red-green-blue). The filters may be different for each channel too.

The red area displays the region of the current filter is computed and is called the receptive field. The number in the filter is called weights or parameters. The filter is sliding (or convolving) around the image, it will compute the element-wise multiplications and fill the output to the output, which is called activation map (or feature map).

We can have multiple convolution layers to identify and capture high-level features, but this comes with more computational power.

Pooling layer

The next layer following the convolution layer is the pooling layer (or down-sampling or sub-sampling). The activation maps are fed into a downsampling layer, this layer will apply one patch at a time (like convolution layer).

The pooling layer progressively reduces the spatial size of the representation. Thus, it reduces the number of parameters and the amount of computational in the network. The pooling layer is also aimed at controlling over-fitting.

Figure 8: Type of pooling layer

There are two main types of pooling layers: max pooling returns the maximum value from the specific pool size (2x2) and average pooling returns the average of all values from the specific pool size, which can be seen from figure 8.

Max pooling performs as a Noise Suppressant, whereas average pooling performs dimensional reduction as noise suppressing mechanism. Hence, max-pooling performs a lot better than the average pooling and is the most common pooling layer to implement.

After going through the above max-pooling layer, we have enabled the ConvNet model to understand the image features. Next part, we will feed it to the fully connected neural network for a classification task.

However, before feeding the output of the pooling layer to the fully connected layer, we need a middle layer to transform the dimension of the data for classification tasks in the FC layer, this middle layer is called the Flatten layer.

Flatten Layer

Figure 9: Flatten layer in action

As mention earlier and the name suggested, this layer will flatten/convert the multi-dimensional arrays into a single long continuous linear vector. In more technical term, it breaks the spatial structure of the data and transforms the multi-dimensional tensor into mono-dimensional tensor, hence a vector.

Dense / Fully connected Layer

Each neuron receives input from all neurons in the previous layer, thus densely connected. The layer has a weight matrix 𝑊, a bias vector 𝛽, and the activation of the previous layer 𝑎. An example can be seen from figure 10a.

Dense implements the operation of: output = activation(dot(input, kernel) + bias) .

Dropout

Figure 10: Dropout method (Source: Giorgio Roffo, Ranking to Learn and Learning to Rank: On the Role of Ranking in Pattern Recognition Applications)

Actually dropout is a regularization method, during training, some of the numbers of layer outputs are randomly ignored (dropped out, switched out). This implementation will have the effect of making layers be treated-like different layers with a different number of nodes and connectivity to the prior layer.

In a convolutional network, dropout is usually implemented in the fully-connected layers and not the convolution layers.