Ultimate Guide to Input shape and Model Complexity in Neural Networks | by Chetana Didugu | Towards Data Science

While building neural networks, a lot of beginners and non-beginners alike, seem to get caught up in figuring out the input shape that needs to be fed into the neural network.
1
2
3
4
5
6
7
8
9
10
Powered by Play.ht
Create audio with Play.ht
Create Audio Narrations with Play.ht
But why should we know the input shape, and why should we feed it? Can’t the Neural Network figure it out on its own?The answer to this question lies in the basics of matrix multiplication.
Suppose we have two matrices A and B. Let the dimensions of B be m rows x n columns. Now, for the two matrices to be compatible for multiplication, the column dimension of A should be the same as the row dimension of B. which means A should be of dimensions k x m, where k can be any number.
Now picture A to be the input tensor (a set of images, a sample set of input features, text data of a particular vocabulary size, etc.) and B to be the first hidden layer in the neural network. k will be the number of input samples, and m is the dimension of each input sample. The shape of m depends on the type of input and the type of hidden layer.
Deep Neural Networks (DNN):
Deep Neural Networks (DNN):
These are the fully connected neural networks that are used for classification and regression tasks. These are also sometimes attached to the end of certain more advance architectures (ResNet50, VGG16, AlexNet, etc.)
Let us look at one such neural network:
model = Sequential()model.add(Dense(units=12, activation='relu', input_shape=(32,)))model.add(Dropout(0.5))model.add(Dense(units=8, activation='relu'))model.add(Dropout(0.5))model.add(Dense(units=6, activation='relu'))model.add(Dropout(0.5))model.add(Dense(units=1, activation='softmax'))
This model consists of three hidden layers and an input layer. Dropout layers are added in between each pair of dense layers for regularisation. The Dropout layer takes and argument “rate”, which specifies the proportion of neurons in the preceding dense layer that should take a value of zero. In this model, the rate is set to 0.5, which means 50% of the neurons in the hidden layers are given a weight of 0.
In Keras, the input dimension needs to be given excluding the batch-size (number of samples). In this neural network, the input shape is given as (32, ). 32 refers to the number of features in each input sample. Instead of not mentioning the batch-size, even a placeholder can be given. Another way to give the input dimension in the above model is (None, 32, ).
If the data is multi-dimensional, like image data, then the input data must be given as (m, n) where m is the height-dimension and n is the width-dimension.
Since 32 is the feature size, it is the column dimension of the input matrix. This means that the row dimension of the hidden layer is also 32. Now that we have the input game sorted, let us look at the model and understand its complexity. Here, by complexity we mean the number of trainable parameters (weight and bias parameters). Higher the number of trainable parameters, more the complexity of the model.
Since each input feature is connected to each neuron in the hidden layer, the total number of connections is the product of the input feature size (m) and the hidden layer size (n). Since each connection is associated with a weight parameter, the number of weight parameters is m x n. Each output neuron is associated with one bias parameter, hence the number of bias parameters is n. The total number of trainable parameters = m x n + n
The first dense/hidden layer has 12 neurons, which is its output dimension. This appears as the second output argument in the model summary, against the first hidden layer. The subsequent dropout layer does not alter the dimension of the output. It only alters the weights of the neurons. The second hidden layer has 8 output neurons, and the next one has 6. The final output layer has 1 neuron. Let us verify the total trainable parameters of this model.
First hidden layer (m = 32, n = 12) : 32 x 12 + 12 = 396Second hidden layer (m = 12, n = 8) : 12 x 8 + 8 = 104Third hidden layer (m = 8, n = 6) : 8 x 6 + 6 =54Output layer (m = 6, n = 1) : 6 x 1 + 1 =7
Total trainable parameters = 396 + 104 + 54 + 7 = 561
2. Convolution Neural Networks (CNN): These are mostly used to process image data for various computer vision applications such as image detection, image classification, semantic segmentation, etc. Since image data is a multi-dimensional data, it requires different types of processing layers that can detect the most important features of the image. This is what convolutional neural networks do.
Images are represented as a tuple of three dimensions — horizontal dimension (width), vertical dimension (height) and the number of channels. If the image is grey-scale, then the channel argument takes a value of 1, and if coloured, then it takes a value of 3, one for each of Red, Green and Blue channels.
Input_shape=(None, 284, 284, 3)cnn=Sequential()cnn.add(Conv2D(16, kernel_size=(2,2), padding="same", activation='relu', input_shape=Input_shape[1:]))cnn.add(MaxPooling2D(4))cnn.add(Conv2D(32, kernel_size=(2,2), padding="same", activation='relu', input_shape=Input_shape[1:]))cnn.add(MaxPooling2D(2))cnn.add(Flatten())cnn.add(Dense(10, activation='softmax'))
Convolutional neural networks have two special types of layers. A convolution layer (Conv2D in the model), and a pooling layer (MaxPooling2D). A 2-D convolution layer of dimension k consists of a k x k filter that is passed over each pixel in the image. Since the k x k filter covers k x k pixels, when it passes over a pixel, k2-1 of its neighbours also get covered. An element-wise multiplication of the filter matrix and the covered image matrix is performed. These values are summed up and populated in the corresponding output pixel.
For example, if the convolution filter of 2 x 2 dimension is passed over an image pixel at position (1, 1), then it covers (0, 0), (0,1) and (1,0) as well. The (0, 0) value of the filter is multiplied with (0, 0) value of the image and so on. We get four values, which are added up and populated into (1, 1) position of the output.
Notice that this shrinks the size of the image. If the filter were to pass over an border pixel say (0, 1), then there would be no output, since this pixel would not have neighbours for the convolution to go through. This would appear as a blacked out border around the image of width k/2.
When a convolution filter of size k x k is passed over an image of size n x n, then the output size becomes n-k+1.
In order to prevent the shrinkage, a padding is added around the border. The argument padding is set to “same”, which means that a padding is added around the image in such a way that the original image size doesn’t change.
The pooling layer does just that; it pools a certain number of pixels in the image and captures the most prominent feature (max pooling) or an aggregate (average pooling) of the pixels as the output. The model contains a Max Pooling layer of size 2 x 2, which captures the maximum pixel value of each 4 pixel cluster. This reduces the size of the output to 1/4 of its original size (or by 1/k2 for a pooling layer of size k x k).
Let us take a closer look at the model. The input dimension is 284 x 284 x 3. It is passed through the first convolution layer of sixteen 2 x 2 filters, with padding. Hence the output dimension of this layer is 284 x 284 x 16. The subsequent layer is a Max Pooling layer of dimension 4 x 4, which shrinks the image by 16 times, 4 times height-wise and 4 times width-wise. So the output dimension is 71 x 71 x 16. The next convolution layer, also with padding, and 32 filters gives an output of 71 x 71 x 32. The next pooling layer of dimension 2 x 2 shrinks the input to dimension 35 x 35 x 32.
This output is fed into an output layer (fully connected/dense layer). However, the data needs to be reshaped into a single dimension before feeding it to the dense layer. This is achieved by the Flatten layer.
For a convolutional layer of m input channels, and n filters (output channels) of filter/kernel size k x k, the kernel goes over each channel of an image individually and produces an output per output channel. Hence for each combination of input-output channel, we need to assign k x k weights. Hence the number of weight parameters is m x k x k x n and the number of biases is equal to the number of channels, n. The total number of parameters is m x k x k x n + n.
A pooling layer does nothing more than aggregate pixel values. Hence there is not trainable parameter here.
Let us verify the tally of model parameters for the convolution neural network.
First Conv layer (m = 3, k = 2, n = 16) : 3 x 2 x 2 x 16 + 16 = 208Second Conv layer (m = 16, k = 2, n = 32) : 16 x 2 x 2 x 32 + 32 = 2080Output layer (m = 39200, n = 10) : 39200 x 10 + 10 = 392010
Total trainable parameters = 208 + 2080 + 392010= 394298
3. Recurrent Neural Networks (RNN): These neural networks are used to process sequential data, or data where the current output depends not only on current input, but also on the previous input. It is used for time series forecasting, natural language processing, etc. A unique feature of RNNs is that it contains gates, which allow or omit input from the previous hidden states to be added to the current input, completely configurable by the user.
There are three main types of RNN. The most basic one is the “vanilla” RNN, which contains one gate. The other two are the Long Short-term Memory units (LSTM), and the Gated Recurrent Unit (GRU). While the LSTM has 4 gates, the GRU has 3, which makes it computationally faster than LSTM units.
Above is an illustration distinguishing the architecture of an LSTM and a GRU, courtesy a detailed blog by Michael Phi, which you can refer to here.
model=Sequential()model.add(LSTM(32, input_shape=(100,2)))model.add(Dense(1))
We have a simple LSTM model (4 gates) here, which feeds into a dense output layer. The model takes an input of three dimensions: batch size, time stamp and features. As is the case with all Keras layers, batch size is not a mandatory argument, but the other two need to be given. In the above example, the input contains 100 time steps and 2 features. Each time step is a sequence of observations (a sequence of words for example). Features are akin to channels in a convolutional neural network. In the model, there are 2 input channels.
In a recurrent neural network with g gates, m input features and n output units, each gate has connections with the current input as well with the hidden state (output) of the previous unit. Hence for each gate, the number of weight parameters is n x n+ m x n. Each output unit has a bias parameter, so the number of bias parameters is n. Total parameter for a single gate is n x n + n x m + n. For g gates the total is g x (n x n + n x m + n).
For the above model, let us verify the parameter tally.
LSTM layer (g = 3, m = 2, n = 32) : 3 x (32 x 32 + 32 x 2 + 32) = 4480Output layer (m = 32, n = 1) : 32 x 1 + 1 = 33
Total trainable parameters = 4480 + 33 = 4513
Knowing the input shape is very important to build a neural network because all the linear algebraic computations are based on matrix dimensions. However, this post attempts to solve the mystery around it. Also, Keras has very comprehensive documentation on each of these models and the input shape, which makes dealing with this challenge a tad bit easier.