Serve a TensorFlow 2 Deep Learning Classifier as a REST API (with GPU support), Part I | by Hamed ZITOUN | Towards Data Science

This series of articles will help you to move from notebooks to real-world use cases of Data Science techniques! We’ll take a Deep Learning classifier from zero to production 🚀!
Disclaimer: You should have prior knowledge and experience with CNN-Convolutional Neural Networks.
The dataset we will use to train our Convolutional Neural Networks classifier is CIFAR-10 (don’t download it yet). The CIFAR-10 dataset comprises 60000 32x32 color images in 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images.
Here are the classes in the dataset, and 10 random images from each:
Ready? Let’s go!
Too much inspired by the cookiecutter-data-science open-source template, here is my favorite Data Science project structure:
In the data/raw, I copy my original data and in data/processed; I keep the pre-processed data.
In the deploy folder, you can find the Dockerfile and either requirements.txt or environment.yml (holding only production libs)
In the models’ folder, I store the saved models.
In the notebooks folder, I have my notebooks (.ipynb files)
In the references folder, I put the metadata and all references to resources I’m using in the project (pdf, notes... etc)
In the reports folder, I put all the graphs, and reports (png, pdf, slides) containing insights to share with my team
In the src folder, there are my Python scripts grouped in the following manner: src/api folder for scripts aiming to create a REST API, src/data folder contains scripts for processing the data, src/dags folder contains my Airflow DAGs, src/models folder for scripts dealing with our ML/DL models and finally, src/visualization folder holds scripts for data viz and reports exporting.
The tests folder contains the python script for testing my app
logging.yaml where I configure my Python logger
db_config.json just a JSON file holding some database access configs
For simplification reason, let us ease off a bit our project structure by only creating (keep everything empty for the moment):
data folder
deploy folder
models folder
notebooks folder
src folder within src/model and src/api sub-folders
environment.yml file
logging.yml file
Just like me, you end up with these folders
Great! You got a clean project structure ready! Let us set up our working environment!
Follow these steps to create a virtual Python 3 work environment defined in environment.yml file and configure your GPU:
2.1 — Install Miniconda 3 (or Anaconda 3)
2.2 — Make sure that conda is on your path (otherwise add the bin file to your $PATH):
$ which conda~/anaconda3/bin/conda
2.3 — If you have a NVIDIA® GPU card with CUDA® Compute Capability 3.5 or higher, enable TensorFlow GPU support by following these steps https://www.tensorflow.org/install/gpu#software_requirements (don’t pip install tensorflow yet! configure CUDA first we’ll install tf2 in the next step 2.4)
2.4 — We now call our virtual environment tf2-image-classifier-api and then specify the required packages in environment.yml file (copy-paste the following code): tensorflow-gpu, numpy, seaborn, and jupyter are for image classifier creation. Flask and waitress are for REST API creation. tqdm for getting a pleasant progress bar and finally, opencv-python is for image pre-processing!
2.5 — Now go to the project root and create a virtual environment:
$ conda env create -f environment.yml
2.6 — Activate the virtual environment
conda activate tf2-image-classifier-api
You should now have your virtual environment activated!
Bonus (FYI only not needed now):
To entirely remove the environment
conda env remove — name tf2-image-classifier-api
To update the environment packages to the latest compatible version
conda env update -f environment.yml
2.7 — Download CIFAR-10 Dataset but in png format from here
2.8 — Copy the downloaded cifar.tgz file to the data/raw folder and unzip it there (you should get the train folder, the test folder and the labels.txt file)
Great! Now your environment and your dataset are ready for our image classifier training and predicting!
Start by running jupyter notebook command in the project’s root:
$ jupyter notebook
Now navigate to http://localhost:8888/tree/notebooks
Create a new notebook
And follow these steps:
For the first time, it would take a few minutes to load the required cuDNN libs (help TensorFlow discover your GPU device). If everything is ok, you’ll get similar results to mine (except that I have a GTX 960M/CUDA v10.2).
But before we need to define two methods; read_image & load_dataset
Read an image using OpenCV
Test the method by visualizing some images
Results
Load dataset and return images and targets as numpy arrays
Now we are ready to load our training & test datasets!
Check your training data & test data shapes
If everything is OK, you’ll get results similar to mine.
Great! Now we have loaded our libs, and our datasets are ready! Let’s create our CNN architecture!
We do not need to re-invent the wheel, we will use this CNN architecture but in TensorFlow 2.2!
The entire model comprises 16 layers in total:
Batch NormalizationConvolution with 64 different filters in size of (3x3)Max Pooling by 2
Batch Normalization
Convolution with 64 different filters in size of (3x3)
Max Pooling by 2
ReLU activation function
Batch Normalization
4. Convolution with 128 different filters in size of (3x3)
5. Max Pooling by 2
ReLU activation function
Batch Normalization
6. Convolution with 256 different filters in size of (3x3)
7. Max Pooling by 2
ReLU activation function
Batch Normalization
8. Convolution with 512 different filters in size of (3x3)
9. Max Pooling by 2
ReLU activation function
Batch Normalization
10. Flattening the 3-D output of the last convolutional operations.
11. Fully Connected Layer with 128 units
Dropout
Batch Normalization
12. Fully Connected Layer with 256 units
Dropout
Batch Normalization
13. Fully Connected Layer with 512 units
Dropout
Batch Normalization
14. Fully Connected Layer with 1024 units
Dropout
Batch Normalization
15. Fully Connected Layer with 10 units (number of image classes)
16. Softmax
Cool! Now we have our helper functions ready to create our CNN!
Now we can define our training hyper-parameters:
Everything is OK now to instantiate our model. Also, let’s check its summary.
That should print
Model: "keras_image_classification_model"_________________________________________________________________Layer (type)                 Output Shape              Param #   =================================================================inputs (InputLayer)          [(None, 32, 32, 3)]       0         _________________________________________________________________batch_normalization_15 (Batc (None, 32, 32, 3)         12        _________________________________________________________________conv_1_features (Conv2D)     (None, 32, 32, 64)        1792      _________________________________________________________________max_pooling2d_8 (MaxPooling2 (None, 16, 16, 64)        0         _________________________________________________________________batch_normalization_16 (Batc (None, 16, 16, 64)        256       _________________________________________________________________conv_2_features (Conv2D)     (None, 16, 16, 128)       73856     _________________________________________________________________max_pooling2d_9 (MaxPooling2 (None, 8, 8, 128)         0         _________________________________________________________________batch_normalization_17 (Batc (None, 8, 8, 128)         512       _________________________________________________________________conv_3_features (Conv2D)     (None, 8, 8, 256)         819456    _________________________________________________________________max_pooling2d_10 (MaxPooling (None, 4, 4, 256)         0         _________________________________________________________________batch_normalization_18 (Batc (None, 4, 4, 256)         1024      _________________________________________________________________conv_4_features (Conv2D)     (None, 4, 4, 512)         3277312   _________________________________________________________________max_pooling2d_11 (MaxPooling (None, 2, 2, 512)         0         _________________________________________________________________batch_normalization_19 (Batc (None, 2, 2, 512)         2048      _________________________________________________________________flatten_2 (Flatten)          (None, 2048)              0         _________________________________________________________________dense_1_features (Dense)     (None, 128)               262272    _________________________________________________________________dropout_4 (Dropout)          (None, 128)               0         _________________________________________________________________batch_normalization_20 (Batc (None, 128)               512       _________________________________________________________________dense_2_features (Dense)     (None, 256)               33024     _________________________________________________________________dropout_5 (Dropout)          (None, 256)               0         _________________________________________________________________batch_normalization_21 (Batc (None, 256)               1024      _________________________________________________________________dense_3_features (Dense)     (None, 512)               131584    _________________________________________________________________dropout_6 (Dropout)          (None, 512)               0         _________________________________________________________________batch_normalization_22 (Batc (None, 512)               2048      _________________________________________________________________dense_4_features (Dense)     (None, 1024)              525312    _________________________________________________________________dropout_7 (Dropout)          (None, 1024)              0         _________________________________________________________________batch_normalization_23 (Batc (None, 1024)              4096      _________________________________________________________________dense_1 (Dense)              (None, 10)                10250     _________________________________________________________________softmax (Softmax)            (None, 10)                0         =================================================================Total params: 5,146,390Trainable params: 5,140,624Non-trainable params: 5,766
Wonderful! 5,140,624 params to train!
To train our model, we will use RMSprop as an optimizer & SparseCategoricalCrossentropy as a loss function. Why SparseCategoricalCrossentropy not just CategoricalCrossentropy? Good question! Here is why!
Now we are good to compile our model!
Everything is ready to train our model. Let us, however, define three callbacks:
EarlyStopping to stop training when no our metric has stopped improving
TensorBoard callback to check our model’s training history
TqdmCallback tqdm keras callback for epoch and batch progress
If everything is OK, your model should start training! You should get results similar to mine.
Great! Your GPU is burning now 🔥! Leave it training your model and we’ll get back when training finishes! (Here a screenshot of my GPU working hard)
Take snap if you want like Tom!
Cool! Our classifier has finished training! Time to evaluate it, right? How is it working on the test images? We can check that by launching the tensorboard interface in our notebook using the following command.
%tensorboard --logdir logs
This command should start tensorboard with accuracy & loss graphs for training and validation:
But wait, accuracy is not a good evaluation metric for a classifier! Let us check precision, recall, and f1 score on each class! We can do that thanks to scikit-learn’s classification_report method:
These are my results:
precision    recall  f1-score   support    airplane       0.84      0.86      0.85      1000  automobile       0.92      0.92      0.92      1000        bird       0.76      0.72      0.74      1000         cat       0.63      0.67      0.65      1000        deer       0.80      0.81      0.80      1000         dog       0.74      0.72      0.73      1000        frog       0.86      0.86      0.86      1000       horse       0.87      0.86      0.86      1000        ship       0.91      0.90      0.91      1000       truck       0.88      0.89      0.88      1000    accuracy                           0.82     10000   macro avg       0.82      0.82      0.82     10000weighted avg       0.82      0.82      0.82     10000
Nice as a first try, right? 0.82 precision & 0.82 recall! We can tune our hyper-parameters and retrain the model to get better results!
Respond to the story with your results! I would be happy to check them! 😊
Let us make an inference test. Giving an image from the test set, let us discover what our model sees!
Result
Nice! Our model is performing pretty well on test data, we can now export it to disk. Our saved model is for making inferences and will get exposed as a REST API!
Finally, let us check if the saved model works fine by loading it!
Wonderful! Wonderful! Wonderful! You have successfully created, trained (on GPU), tested, and saved a TensorFlow 2 Deep Learning Image classifier!
Now we are ready to expose it as a REST API with Flask and Waitress and later Dockerize everything!
Ready? Go!
But wait, this is already too much for one Medium story 😔! Stay tuned for PART II! 🚀
All the source code is on my GitHub 👇 I’ll keep pushing everything!
github.com
Great! Thank you for reading! I’m eager to see you in Part II (coming soon)!
I hope I’ve helped you create a Deep Learning Image Classifier using TensorFlow 2.2. In Part II, we will expose our saved model as a REST API and later we’ll dockerize everything!
Please consider sharing the story if you enjoyed it. That would help a lot.