Semantic Segmentation using a Django API — DeepLabV3 | by Aymane Hachcham | Towards Data Science

Image segmentation has been a hot topic for a while now. Various uses cases involving segmentation had emerged in a bunch of different areas, machine vision, medical imaging, object detection, recognition tasks, traffic control systems, video surveillance and, a lot more. The intuition behind these intelligent systems is to capture the diverse components that form the image and therefore, teach computer vision models to grasp more insight and a better understanding of the scene and the context.
The two types of image segmentation commonly used are:
Semantic Segmentation: identifying different classes in the image and segment accordingly
Instance Segmentation: determine priorly the different classes in the image and recognize the number of instances each class contains. The image is decomposed in multiple labeled regions relating to the different class instances the model was trained on.
For this article, I will use the Pytorch implementation of the Google DeepLab V3 segmentation model to customize the background of an image. The intention is to segment the foreground and detach it from the rest while replacing the remaining background with a whole different picture. The model will be served through a Django REST API.
A bit of background on DeepLab V3Use the DeepLab V3-Resnet101 implementation from PytorchSet up the Django APIConclusion
A bit of background on DeepLab V3
Use the DeepLab V3-Resnet101 implementation from Pytorch
Set up the Django API
Conclusion
You can check the entire code for this project under my Gihut repo.
github.com
Segmentation models use fully convolutional neural networks FCNN during a prior image detection stage where masks and boundaries are put in place then, the inputs are processed through a vastly deep network where the accumulated convolutions and poolings cause the image to importantly decrease its resolution and quality, hence results are yield with a high loss of information. DeepLab models address the challenge leveraging on Atrous convolutions and Atrous Spatial Pyramid Pooling (ASPP) architectures.
Four main stages are involved during the forward process in the DeepLab architecture:
Extract the image features with the backbone models. The backbone used in our case is the Resnet 101 object detection model, serving as a first convolution pipe to catch and mask the important feature maps.
To control the size of the output feature maps Atrous convolutions are used in the last few layers of the backbone.
In the last stages, ASPP architectures classify the different pixels of the output image and process it through 1 x 1 convolution layers to recover their original size.
Let’s kick off the process by creating a Pytorch module that wraps the original DeepLab V3 model. Originally, the Pytorch team already propose their implementation of Google DeepLab V3 architecture pre-trained on the COCO dataset along with various backbones to choose from. For our specific task, we will go with the deeplabv3-resnet101 pre-trained module easily loadable from torchvision.models.segmentation API package.
Set up your python virtual environment using pip virtualenv and install all the required packages:
i. Specify the path of your python virtual env:
virtualenv absolute/path/to/virtual/env/pytorch-en
ii. Activate your virtual env:
source /absolute/path/pytorch-env/bin/activate
iii. Install the required libraries:
pip install torch torchvision
pip install opencv-python
pip install numpy
pip install Pillow=2.2.1
Before coding the Pytorch wrapper, you will need to define a Pytorch dataset class to sample the input image file into a high-level object with multiple methods. That way, the model will be dealing with sampled objects instead of converting and handling all the related file stuff.
def __init__(self, root_dir, image_file, device) The init method takes care of converting the image file into a Pillowimage object and then into a torch tensor, defining internally a set of preprocessing rules to transform the initial tensor into a more suitable input for the deeplabv3 model. The purpose is to generate a well-shaped tensor, dimension wise, to avoid any kind of annoying mismatching between the image files and the model.
The SemanticSeg(nn.Module)wrapper module has three main methods:
def __init__(self, pretrained, device) Initialization with pre-trained value to load an already trained deeplabv3-resnet101 module, and device argument to specify CPU or GPU acceleration when dealing with inference.
def forwar(self, input) Applying module inference on a SegmentationSample input, and returning a tensor of predictions.
def load_model(self, pretrained=False) Load the deeplabv3-resnet101 module from torchvision.models.segmentation API.
Add a method to help process the outcome. Keep in mind that the output tensors have 21 channels matching the prediction results for each target class the model was trained on. Accordingly, we need to decode the tensor shape to be able to output a proper image result.
background_custom(self, input_image, source, background_source, channel=21) The method takes the output tensor image with [1, 21, H, W] shape, the path to the image file, the path to the background image file, and the number of channels that are already set to 21. The goal is to extract the person channel (class 15) from the output, exclude all the remaining channels tagging them as the background, and finally merge the defined background with a new image source.
We will be using the Django REST Framework to build a simple API for serving our model. The key idea is to configure all the required files, including the models, routing pipes, and views so that we can easily test the inference through an easy forward POST and GET requests.
You can also follow the amazing tutorial by Bennett Garner that covers in detail all the required steps to get your API running.
medium.com
Typically, an API is a window into a database. The API backend handles querying the database and formatting the response. What you receive is a static response, usually in JSON format, of whatever resource you requested.
Install djangoand djangorestframework : pip install django pip install djangorestframework .
Head up to the location you will be working on and start your project: django-admin startproject semanticsegmentation .
Within semanticsegmentation/ folder, you will find a manage.py script to run your localhost project: python manage.py runserver .
On 127.0.0.1:8080 port you should land on the welcoming Django page:
From now on your Django backend is correctly set up and running.
You will generate a new application specifically dedicated to the API management inside the previous folder.
Run the following command to create the folder: python manage.py startapp api
Register the new API app with the project folder: Go to semanticsegmentation/settings.py and add the path to the API,
INSTALLED_APPS = ['api.apps.ApiConfig','django.contrib.admin','django.contrib.auth',...]
Now in order to save and retrieve objects from the Django database, you need to build a model class that represents the entity we will be going to process in the API requests. For our particular case, we need to post images, apply the model inference to get the semantic filter and, then recover them back. Consequently, the best fit would be a class with two models.FileFieldattributes to upload the input and output image files to the server, a specific id models.UUIDFieldlinked to each stored image, a name CharFieldto identify them, and optionally, a DateTimeFieldto save the exact time when they were stored.
Define in a separate script the methods that handle the file upload logic:
get_input_image_path
get_output_image_path
Migrate your changes to the database: python manage.py makemigrations
python manage.py migrate
Define the serializers that match the model class, one serializer for the input image and another for the output result:
Register your new model with the admin site: admin.site.register(ImageSegmentation)
We will be creating simple get and post methods that handle simple operations:
POST api_view: It sends two file images, processes them applying the model inference, and saves them to their corresponding input/output folders.
GET api_view: Retrieve the saved image and serve them as static resources.
Set up the url patterns in the semanticsegmentation/urls.py file:
Set up the url patterns in the semanticsegmentation/urls.py file:
urlpatterns = [   path('admin', admin.site.urls),   path(r'bg_custom_api', include('API.urls'))]
2. define the address of your API endpoints in the api.urls file:
urlurlpatterns = [ path(r'test/', views.test_api, name='test_api_communication'), path(r'images/', views.get_images, name='get_images'), path(r'api/', views.run_inference, name='run_inference_on_images'),]
Download Postman and start testing your API locally.
Run Grayscale Inference on 127.0.0.1:8080/api/grayscale/
Run Background custom inference on 127.0.0.1:8080/api/inference/
Once you have set up all the required components for the API to run correctly, you can start experimenting with some really cool features with DeepLab V3, ranging from background custom, background grayscale, background blur and, much other creative stuff I’m sure you can come up with.
A natural step after building the API for model serving would be to develop a small mobile application as a client that interacts with Django backend, hence you can test and fetch all the results previously seen.I’m willing to consider this option for my next article, so stay tuned for the second part.
If you have any questions regarding the code, please get in touch with me and, don’t hesitate to e-mail me at aymanehachchaming@gmail.com
The whole project code for this article can be found under my GitHub repo:
github.com
DeepLabV3 model with a ResNet-101 backbone by Pytorch team
[2017 arXiv] [DeepLabv3] Rethinking Atrous Convolution for Semantic Image Segmentation by the Google Team
DeepLab V3 Atrous convolutions by Sik-Ho Tsang
Credit to Ahmed Barbouche, for his help.