Comparing the Performance of Fully-Connected, Simple CNN, and ResNet50 for Binary Image Classification in TensorFlow | by Binh Phan | Towards Data Science

In the world of machine learning, there are three models that you can use to perform binary image classification: a fully-connected network, a convolutional neural network, or a pre-trained network like MobileNet with transfer learning applied to it. In my previous stories, I showed how to implement these models to build an image classifier that can classify dandelions and grass and evaluate their performance by computing their accuracy, ROC curves, and AUC score. In this story, I will compare these models side-by-side and compare their performances. If you’d like to follow along the entire training, evaluation, and comparison of these models, follow this notebook. You can run the code by saving your own copy by going to File -> Save a copy.
Which model do you think will perform best?
Firstly, let’s take a look at the three model architectures and see how they differ:
FULLY-CONNECTED MODELLayer (type)                 Output Shape              Param #    ================================================================= flatten (Flatten)            (None, 120000)            0          _________________________________________________________________ dense (Dense)                (None, 128)               15360128   _________________________________________________________________ dense_1 (Dense)              (None, 1)                 129        ================================================================= Total params: 15,360,257 Trainable params: 15,360,257 Non-trainable params: 0CONVOLUTIONAL NEURAL NETWORK MODELLayer (type)                 Output Shape              Param #    ================================================================= conv2d (Conv2D)              (None, 198, 198, 16)      448        _________________________________________________________________ max_pooling2d (MaxPooling2D) (None, 99, 99, 16)        0          _________________________________________________________________ conv2d_1 (Conv2D)            (None, 97, 97, 32)        4640       _________________________________________________________________ max_pooling2d_1 (MaxPooling2 (None, 48, 48, 32)        0          _________________________________________________________________ conv2d_2 (Conv2D)            (None, 46, 46, 64)        18496      _________________________________________________________________ max_pooling2d_2 (MaxPooling2 (None, 23, 23, 64)        0          _________________________________________________________________ conv2d_3 (Conv2D)            (None, 21, 21, 64)        36928      _________________________________________________________________ max_pooling2d_3 (MaxPooling2 (None, 10, 10, 64)        0          _________________________________________________________________ conv2d_4 (Conv2D)            (None, 8, 8, 64)          36928      _________________________________________________________________ max_pooling2d_4 (MaxPooling2 (None, 4, 4, 64)          0          _________________________________________________________________ flatten (Flatten)            (None, 1024)              0          _________________________________________________________________ dense (Dense)                (None, 512)               524800     _________________________________________________________________ dense_1 (Dense)              (None, 1)                 513        ================================================================= Total params: 622,753 Trainable params: 622,753 Non-trainable params: 0MOBILENET w/ TRANSFER LEARNINGLayer (type)                 Output Shape              Param #    ================================================================= mobilenetv2_1.00_224 (Model) (None, 7, 7, 1280)        2257984    _________________________________________________________________ global_average_pooling2d (Gl (None, 1280)              0          _________________________________________________________________ dense (Dense)                (None, 1)                 1281       ================================================================= Total params: 2,259,265 Trainable params: 1,281 Non-trainable params: 2,257,984
Notice how the fully-connected model has the highest number of trainable parameters, followed by the CNN model, followed by the MobileNet model. In the FC model, every connection in the Flatten layer is connected to every connection in the Dense layer, resulting in a higher number of parameters. In the CNN, fewer parameters are needed because every convolutional layer reduces the dimensions of the input through the convolution operation. And in contrast to those two models, the MobileNet model uses what is called transfer learning: it doesn’t train on the pre-trained layers and only trains a final Dense layer. It takes advantage of the feature extraction capabilities of the pre-trained layers for training that last layer, so only few parameters are needed to train the model.
For each model, I used the same dataset and trained the model with 15 epochs. Let’s bring the results together and compare them side-by-side, starting with the accuracies :
FC accuracy: 0.5987CNN accuracy: 0.7197MobileNet accuracy: 0.8917
The MobileNet model has the highest accuracy, followed by the CNN model, followed by the fully-connected model.
Accuracy isn’t the best way to compare model performance. Better metrics for classification tasks include ROC curves and AUC, which measure how much our model is capable of distinguishing between our two classes, dandelions and grass. The higher the AUC, the better our model is at classification. In the plot above, MobileNet scores highest in AUC, followed by CNN, followed by FC.
Here are the takeaways from these comparisons:
The MobileNet model is a superior model in terms of accuracy and ROC curve/AUC, even though it doesn’t have the most parameters. This shows the power of transfer learning over a powerful trained model.
The CNN model comes in 2nd. Convolutional Neural Networks perform better than fully-connected networks on binary image classification, with a lot less parameters, because of their shared-weights architecture and translation invariance characteristics.
The fully-connected model performs better than chance at binary image classification, but it isn’t the optimal model to do so.