Guide to Advanced CNNs in tensorflow | Towards Data Science

The first ConvNet model I built was to identify dogs or cats and had a basic structure of a sequential model having Conv2d layers along with a mix of batch normalization and max-pooling layers here and there and not to forget a dropout layer to counter overfitting😏. All this was followed by a flattening layer which ultimately led to a Dense layer. They were added to the model using add. It worked well and gave a good enough accuracy. Then I tried the famous MNIST dataset and used the same architecture. All I had to do was to change the loss in my dense layer from binary cross-entropy to categorical cross-entropy and tada I felt I had mastered CNNs. But, boy, was I wrong! There is a lot more to it like making sparsely connected architectures with different types of Conv2d layers, applying different techniques for overfitting and lots more.
Advanced Conv2D layers
Advanced Conv2D layers
Depthwise Separable Convolutions
Dilated Convolutions
2. Counter Overfitting
Spatial Dropout
Gaussian Dropout
Activity Regularization
3. Complex Architectures
Sparsely Connected Architectures
Skip Connections
4. Conclusion
They are more efficient, require less memory and less computation and can even provide better results in some cases. Due to these characteristics, they are generally used when our model has to be deployed on Edge/IoT devices because they have limited CPU and RAM. It divides the process of a normal convolution layer into two processes i.e. depthwise convolution and pointwise convolution. In depthwise convolution, the kernel iterates over one channel at a time. The pointwise convolution uses a 1x1 kernel to increase the number of channels. This way the total number of multiplications required is reduced and that makes our network faster. This is a great article to learn more about it.
tensorflow.keras.layers.SeparableConv2D(32, (3, 3), padding="same"))
Dilated convolutions can be implemented in normal convolution layers as well as depthwise separable convolution layers. It is a normal convolution operation with gaps. Along with providing a larger receptive field, efficient computation and lesser memory consumption it also preserves the resolution and order of data. Hence it generally improves the performance of the model.
#In Normal Conv2Dtf.keras.layers.Conv2D(32, (3, 3), padding='same', dilation_rate=(2, 2))#In Seperable Conv2Dtf.keras.layers.SeparableConv2D(no_of_filters, (3, 3), padding='same', dilation_rate=(2, 2))
In normal dropout, some randomly selected neurons are ignored during training. Those neurons do not contribute in the forward pass and their weights are not updated during backward pass. This leads the network to learn multiple independent internal representations and it makes it less probable to overfit on the training data. In spatial dropout instead of dropping neurons, we drop out entire feature maps. As stated in the Keras documentation for spatial documentation:
If adjacent pixels within feature maps are strongly correlated (as is normally the case in early convolution layers) then regular dropout will not regularize the activations and will otherwise just result in an effective learning rate decrease. In this case, spatial dropout will help promote independence between feature maps and should be used instead.
tf.keras.layers.SpatialDropout2D(0.5)
It is a combination of dropout and Gaussian noise. That means that this layer along with dropping some neurons also applies multiplicative 1-centered Gaussian noise. Like the normal dropout, it also takes the argument rate. From its documentation:
Float, drop probability (as with dropout). The multiplicative noise will have standard deviation sqrt(rate / (1 — rate)).
For more in-depth reading about it, you can refer to this article here.
tf.keras.layers.GaussianDropout(0.5)
Regularization makes slight changes to networks so that they generalize better. It encourages a neural network to learn sparse features or internal representations of raw observations which makes the model perform better on unseen data. There are three types of regularization techniques supported:
l1: Activity is calculated as the sum of absolute values.
l2: Activity is calculated as the sum of the squared values.
l1_l2: Activity is calculated as the sum of absolute and sum of the squared values.
The value of weight matrices decreases due to the addition of the regularization term and it leads to simpler models which reduces overfitting. For more information about regularization techniques refer here.
#L1 regularizationtf.keras.layers.ActivityRegularization(l1=0.001)#L2 regularizatontf.keras.layers.ActivityRegularization(l2=0.001)#L1_L2 regularizationtf.keras.layers.ActivityRegularization(l1=0.001, l2=0.001)
There are two types of architectures, densely connected and sparsely connected.
It was popularised by the Inception network. So what was the need for these types of architectures? Can’t we just keep on adding layers and the deeper the network the better the result? Well, not at all. The bigger the model, the more prone is it to overfitting especially with smaller datasets and it also increases the amount of computation power required to train it. Sometimes even the training error even becomes worse. Sparsely connected architectures help us in increasing the depth and width of our models while not overshooting the computation power needed. They can have different sizes of kernels for convolutions which also helps the model when the test objects are of different sizes.
Generally, we use tf.keras.layers.add to make densely connected architectures. We can visualize our models using tf.keras.utils.plot_model.
from tensorflow.keras.models import Sequentialfrom tensorflow.keras.layers import Conv2D, add, Inputfrom tensorflow.keras.utils import plot_modelmodel = Sequential()model.add(Conv2D(64, (3, 3), activation = 'relu', padding='same', input_shape=(128, 128, 3)))model.add(Conv2D(64, (3, 3), activation = 'relu', padding='same'))model.add(Conv2D(128, (3, 3), activation = 'relu', padding='same'))plot_model(model)
We can also create models like these by assigning them to variables and placing them the variable name to the layer it is connected to at the end. Input shape is specified in tf.keras.layers.Input and tf.keras.models.Model is used to underline the inputs and outputs i.e. the first and last layer of our model.
from tensorflow.keras.models import Modelfrom tensorflow.keras.layers import Conv2D, Inputfrom tensorflow.keras.utils import plot_modelinput_flow = Input(shape=(128, 128, 3))x = Conv2D(64, (3, 3), activation = 'relu', padding='same')(input_flow)x = Conv2D(64, (3, 3), activation = 'relu', padding='same')(x)x = Conv2D(128, (3, 3), activation = 'relu', padding='same')(x)model = Model(inputs=input_flow, outputs=x)plot_model(model)
Now, to make sparsely connected architectures we just need to assign some layer to multiple layers and then use tf.keras.layers.concatenate to join them.
from tensorflow.keras.models import Modelfrom tensorflow.keras.layers import Conv2D, Input, concatenatefrom tensorflow.keras.utils import plot_modelinput_flow = Input(shape=(128, 128, 3))x = Conv2D(64, (3, 3), activation='relu', padding='same', input_shape=(128, 128, 3))(input_flow)one = Conv2D(64, (3, 3), activation='relu', padding='same')(x)two = Conv2D(64, (5, 5), activation='relu', padding='same')(x)three = Conv2D(64, (7, 7), activation='relu', padding='same')(x)x = concatenate([one, two, three])x = Conv2D(128, (3, 3), activation = 'relu', padding='same')(x)model = Model(inputs=input_flow, outputs=x)plot_model(model)
This was popularised by the network ResNet50. The main idea behind skip connections was to solve this problem given below by Microsoft Research as given in their paper Deep Residual Learning for Image Recognition.
When deeper networks are able to start converging, a degradation problem has been exposed: with the network depth increasing, accuracy gets saturated (which might be unsurprising) and then degrades rapidly. Unexpectedly, such degradation is not caused by overfitting, and adding more layers to a suitably deep model leads to higher training error...
Its advantage is that it makes estimating good values for the weights easier and the model generalizes better.
If you understood how to make sparsely connected architectures, you can easily guess how this will work but in this case to join them we will use tf.keras.layers.Add.
from tensorflow.keras.models import Modelfrom tensorflow.keras.layers import Conv2D, Input, Addfrom tensorflow.keras.utils import plot_modelinput_flow = Input(shape=(128, 128, 3))x = Conv2D(64, (3, 3), activation='relu', padding='same', input_shape=(128, 128, 3))(input_flow)one = Conv2D(64, (3, 3), activation='relu', padding='same')(x)two = Conv2D(64, (5, 5), activation='relu', padding='same')(one)three = Conv2D(64, (7, 7), activation='relu', padding='same')(two)x = Add()([x, three])x = Conv2D(128, (3, 3), activation = 'relu', padding='same')(x)model = Model(inputs=input_flow, outputs=x)plot_model(model)
Using these techniques might improve our model, but that’s not always the case. They might even perform worse than a simple densely connected architecture with normal Conv2D layers. You will need to try and see which strategy gives the best results.
For revision purposes, I have created a model using the concepts provided above. This is just for revision purpose and I do not garuntee that this model will give better results.