Full stack PyTorch Crowd Size Estimator | by Charlie Mackie | Towards Data Science

Please try the demo of our crowd size estimator: https://app3-qrhrzckmpq-ue.a.run.app
You can choose from some of the images in the ShanghaiTech dataset, or you can upload your own image. If you choose to upload your own image it must be a JPG/JPEG and under 100KB.
You can check the image size as follows:
Computer:
Windows: Explorer → Select Image (bottom of the window).
Mac: Control + Click → ‘Get Info’ → ‘Size’
Mobile:
iPhone: Ensure the image is not a screenshot, then you can “Choose Size” of the image (bottom of screen). Choose “Small” and upload.
Android: Check out this blog post, https://www.quora.com/How-do-I-reduce-the-size-of-photos-in-Android
In this article, I will explain how my Western AI project team developed a web application that estimates the size of a crowd in a given image.
Our goal from September 2019 was to deploy a machine learning model that could predict the sizes of crowds. We accomplished our goal and we were able to showcase our project at the Canadian Undergraduate Conference on AI (CUCAI) hosted by QMIND — Queen’s AI Hub.
Crowd size estimation is a popular topic in the machine learning community. It has various applications including crowd control, customer management, and urban planning. Our model could be applicable in a school setting, measuring class attendance or building occupancy.
Pictured above is our deployed web application. The web host takes a local image from the user in a JPG format then estimates the size (number of people) of the crowd.
The purpose of this article is to:
Showcase our project: Size.AI
Give insight into crowd recognition and machine learning as they stand today
Provide a tutorial for the deployment of a machine learning model
The article will be structured as follows:
Brief introduction to machine learning, computer vision, and crowd size estimation
Current crowd size estimation approaches
Our approach: CSRNet
Training, testing, and validation (Google Colab/Python3)
Deployment (Flask/Google Cloud)
Next steps, sources, and Western AI
Artificial intelligence (AI) and machine learning (ML) have different meanings. Machine learning is a type of AI that involves statistical analysis and classification of problems. Popular machine learning applications include Natural Language Processing (NLP) and Computer Vision (CV). These applications are powered by machine learning algorithms which include K Nearest Neighbours (kNN), regression, and neural networks.
Let’s focus on neural networks. Neural networks are loosely inspired by the structure of our brain. In the brain there is a network of neurons and each neuron stores bits of information. The neurons are attached to each other by axons which can transfer signals between neurons. In an artificial neural network, the neurons are represented by nodes which are organized into layers.
The first layer of nodes can take some form of input and each layer proceeding will help classify the input. The last layer represents an easily interpretable output.
Example: The computer wants to determine whether an image contains a dog or a cat. The image will be broken down to its pixels, and each pixel will be represented by an RGB Scale Value (0–255). The RGB values are inputs to the first layer of nodes. Each layer after that can represent some type of feature detection within the image; superposition of the eyes, head size compared to ear size ... etc. The final layer of nodes aggregates all of the information from previous layers and makes a decision.
The ‘learning’ part of machine learning, involves iterating through many examples that are tagged with metadata which has already been classified by humans. The neural network undergoes the process of back-propagation where the error of the network is minimized. For each example provided, the network will make its prediction, then validate its prediction with the metadata provided and come up with a cost of error (predicted - actual). The goal of back-propagation is to minimize this error by adjusting the network.
The math behind this is more complicated than I just explained. If you are still interested, check out Jayson’s (Project Manager), Simple classifier intro to neural networks.
Crowd size estimation uses neural networks to classify people in a crowd then aggregate the amount of people detected. Currently there are three approaches to crowd size estimation: Detection, Regression, and Density.
Detection: Focuses on object detection; recognizing features of a human (Ex: head, shoulders, body ... etc). This approach is typically more accurate for smaller crowds of people.
Regression: Uses broader features of a person and puts more focus on edge details. This outperforms the detection models when presented with more dense crowds.
Density: (See density map example below) Instead of focusing on individual people, density maps are produced that track groups of people. This is the most accurate method for dense crowds, and the method used by our model of choice: CSRNet.
Congested Scene Recognition Net (CSRNet) is a crowd size estimation model that was proposed by computer scientists at the University of Illinois Urbana-Champaign in 2018.
CSRNet leverages the VGG-16 image classification network as the frontend portion since it has a flexible and effective architecture. The VGG-16 uses a variety of activation functions (used to abstract node outputs inter-layer) and other hyper-parameters (layer/node configuration) including ReLU and SoftMax. ReLU is standard in machine learning, it means Rectified Linear Unit. SoftMax is less common but noteworthy. The goal of SoftMax is to turn numbers into probabilities and polarize classification. This can be useful in image classification at the last layer because it provides probabilities for any classification scenario. Since our goal is to output a density map instead of one-hot classification, CSRNet’s backend structure replaces the fully connected and SoftMax layers with more convolutional layers.
Dilated Kernels: In the backend, dilated convolutional layers are used in place of max pooling in order to retain the output dimensions and increase the receptive field. The dilated kernel is a standard convolutional kernel that is dispersed by a dilation factor (seen below). Dilated kernels do not reduce image quality which is important because the larger the density map dimensions, the more accurate our estimate can be.
The dilated convolutions help produce our density maps:
Results: CSRNet achieved better MAE and MSE results than the other popular Crowd Size Estimation Networks.
For some more context, MAE is the Mean Absolute Error. Mean absolute error is the average of the absolute error for every training image. MSE is the Mean Squared Error. MSE is different than MAE because it puts more emphasis on large errors.
Now that all the Queen’s students are gone — let’s code! We used a tutorial from Analytics Vidhya: Building your own Crowd Counting Model . We stored all of our pre-processing, training, testing, and validation scripts in a Google Drive folder. In Google Drive we were able to employ Google Colab Notebooks to create and share our model. All scripts were written and compiled in Python 3 or Python 2 in some cases. We also installed a CUDA GPU in order to train our model.
We used the ShanghaiTech dataset because of its breadth in camera perspective, variety of crowd sizes, and large quantity of annotated data. The dataset contains almost 1200 images with over 330,000 annotated heads.
The end goal of the training and testing in Google Drive was to produce our trained Model.PT file (PyTorch file) that we could upload to our front-end interface. In order to reach that goal, we had to compile and run:
Ground truth production script
Training script
Validation script
Ground truth production script: The purpose of this script was to convert the ground-truths provided by the dataset into density maps that could be used for training.
Below you can see and example of the output. Converting the ground truth into a density map.
Show the raw predicted count from a density map:
np.sum(groundtruth) # Output -> 166.15
Training Script: The training script iterated through all of our train data set images and improved accuracy between each Epoch (Epoch is an iteration through the data).
Example code of running the training script with our first data set:
root = "/content/gdrive/My Drive/Western AI - Intro to Computer Vision Project/CSRNet Crowd Counting Model/Data/"!python2 '/content/gdrive/My Drive/Western AI - Intro to Computer Vision Project/CSRNet Crowd Counting Model/train.py' '/content/gdrive/My Drive/Western AI - Intro to Computer Vision Project/CSRNet Crowd Counting Model/part_A_train.json' '/content/gdrive/My Drive/Western AI - Intro to Computer Vision Project/CSRNet Crowd Counting Model/part_A_test.json' 0 0
Here is a snapshot of one of the training Epochs (15th):
We were able to train our model for 100 full epochs, time and resource permitting. While monitoring the training we noticed a slight improvement in MAE with each epoch. If you look at the output above, the MAE in the 15th epoch was 70.621. That means the average error was 70.621 people (+/-). The MAE in the first epoch was 250, so we made some progress. With the size of the dataset (1200 images) and the aggregated size of every crowd (330,000 heads), there was an average of 275 people in each crowd. Given the large crowd sizes, our errors are satisfactory.
Validation Script: This script took our trained model and determined the MAE and percent error over every testing example. (Note: this is an abstracted version of the validation script)
checkpoint = torch.load('/content/gdrive/My Drive/Western AI - Intro to Computer Vision Project/CSRNet Crowd Counting Model/model.pt')model.load_state_dict(checkpoint['state_dict'])import numpy as npmae = []for i in xrange(len(img_paths)):    ...    mae.append(abs(np.sum(groundtruth)))    print i, mae[i]
The results varied, but finished with a MAE average of 90.19 for testing. This correlates well with the MAE values from the training script and does not indicate any significant overfitting/underfitting of our training set. Here is an snippet of some of the results:
We ran a similar variation of this validation script that gave us model error/accuracy. The MAE values above correspond to a total testing error of 15.3% (two test sets: A and B). This means that our model had 84.7% accuracy.
Model.pt: Once trained and tested we exported our model to a .pt file (PyTorch) that we could deploy on Flask. The model.pt file contains all the network parameters and weights stored in a massive dictionary object.
You can check out the GitHub Repository for our Flask deployment here: https://github.com/jaysondale/Size.AI-Deployment
Flask is a Python web framework built with an easy-to-extend philosophy (https://www.fullstackpython.com/flask.html). Launching your own web application is as easy running an ‘app.py’ Python file with this structure:
from flask import Flaskapp = Flask(__name__)@app.route('/')def hello_world():    return 'Hello, World!'if __name__ == '__main__':    app.run()
In our deployment framework there are 5 key components:
App— launches our localhost port 5000
Inferences— initializes our model and makes the predictions
Commons —references our model file
Model.pt — our trained model
Templates— contains index and result templates (render results page)
App (‘app.py’): the hello_world function (above) returns the content and functionality to be displayed in the web app. In our case we needed to accept an image from the user and return a prediction. When dealing with user input, we need to use HTTP Methods. ‘GET’ renders the template that we desire, ‘POST’ registers a user that can perform actions (upload a file). Here you can see the ‘upload_file’ function in the ‘app.py’ file:
def upload_file():    if request.method == 'POST':       absolute_path = os.path.abspath("../")       if 'file' not in request.files:          return redirect(request.url)       file = request.files['file']       if not file:          return       print("GETTING PREDICTION")       filename = secure_filename(file.filename)       file.save(os.path.join(app.config['UPLOAD_FOLDER'],filename))       prediction = get_prediction(file)        return render_template('result.html', Prediction=prediction,        File=filename)        return render_template('index.html')
Inferences (‘inference.py’): initializes our model and gets a prediction from it. The App script sent a file from the user as a parameter. The get_prediction function sends the image to the model and outputs the result:
model = get_model() # initialize modeldef get_prediction(file):     img = transform(Image.open(file).convert('RGB')).cpu()     output = model(img.unsqueeze(0))    prediction=int(output.detach().cpu().sum().numpy()) # prediction    print("Predicted Count: ",int(output.detach().cpu().sum().numpy()))     return prediction
Commons (‘commons.py’): references our model.pt file. We import and initialize CSRNet from PyTorch, then we load our state dictionary (our weights and biases from training):
def get_model():    model = CSRNet() # original CSRNet    model.load_state_dict( torch.load('/Users/charliemackie/CSRNET_DEPLOYMENT/SIZE.AI/Pytorch/model (2).pt', map_location='cpu')) # local reference to model.pt    model.eval()    return model
The ‘/Users/charliemackie/CSRNET_DEPLOYMENT/SIZE.AI/Pytorch/model (2).pt’ is the path to the Model.pt saved in our project directory.
Templates: has our custom HTML and CSS templates that will display our User Interface. These files have a combination of text labels, buttons, and titles. The most important part is displaying our prediction which is embedded in a text body like this:
<h2 class="h5 mb-3 font-weight-normal"><i>{{Prediction}}</i></h2>
With all of these components working together, we have our functioning Flask application!
Our deployment process utilized three core Google Cloud features: Cloud Storage, Build, and Run. The first step in creating a cloud-based web system was to containerize our flask application, meaning it would contain all requirements and commands to run on any given computer system. While running, the cloud platform is programmed to listen on a specific network port at the IP address 0.0.0.0:8080. This IP and port was directly hardcoded into the flask app. The following files were added to the flask application directory to complete the containerization process:
Requirements (“Requirements.txt”): This file contains all of the required python modules for the application to run.
Docker (“Dockerfile”): This file contains a series of CLI commands which initiate the flask app.
Dockerfile Contents:
FROM python:3RUN apt-get update -yRUN apt-get install -y python-pip python-dev build-essentialCOPY . /appWORKDIR /appRUN pip install -r requirements.txtENTRYPOINT [“python”]CMD [“app.py”]
YAML (“Cloudbuild.yaml” ): This file is used with the Google Cloud command line interface (CLI) and provides additional build parameters (this includes the build destination).
Yaml file contents:
steps:- name: ‘gcr.io/cloud-builders/docker’args: [‘build’, ‘-t’, ‘gcr.io/size-ai/app’, ‘.’]- name: ‘gcr.io/cloud-builders/docker’args: [‘push’, ‘gcr.io/size-ai/app’]images: [‘gcr.io/size-ai/app’]
Once these files were added, the containerized application needed to be compressed and uploaded to Google Cloud Storage where it could be used in a web service. Once uploaded, Google Cloud Build would follow the steps provided by cloudbuild.yaml and construct a cloud-based application that would be ready for deployment. The following command was executed in the size-deployment directory to accomplish both of these tasks.
$ gcloud builds submit
By default, the gcloud interface looks for “cloudbuild.yaml” and completes the upload and build procedure. Once built, A web service was created in Google Cloud Run to host the containerized application. When configuring the service, we used the maximum memory and timeout allowances of 2GiB and 900s to give users the opportunity to upload the highest quality images possible. Once the build revision was selected, our application was successfully deployed on Google Cloud!
The Google Cloud dashboard (below) lets us analyze many useful metrics including: Request count, Request latencies, Container CPU utilization, and Container memory utilization. With the ‘Request count’ metric, we can view every instance of our app that has been run.
Next Steps:
Tune hyper-parameters: We want to try and tune the network to achieve better results. Hyper-parameters could mean number of layers, layer size, or activation functions. Currently we run an off-the-shelf VGG-16/CSRNet structure — there are lots of possibilities for modification. Some areas that we have highlighted include adjusting the learning rate and/or using a longer training period on an enhanced GPU. We want to test different learning rates on a logarithmic scale in the near future.
Optimizer choice: Our model currently uses a Stochastic Gradient Descent (SGD) optimizer. This means the mini-batch size is 1 and gradient descent is performed after each training example. We have discussed experimenting with an Adam optimizer or RMSprop. Once we implement a new optimizer we will have more hyperparameters to tune including adjusting our learning rate.
Deploy at a larger Scale: Our web application is currently limited because of the size of our model. One potential solution is to use transfer learning to train a more efficient version of the model. We were inspired by Geoffrey Hinton when he presented to us at CUCAI re: Transfer Learning to develop light weight ML models. If we accomplish this, we could look into publishing an IOS app.
Western AI:
Western AI is the first student-led organization at Western that is aiming to create a community for Artificial Intelligence on campus. This means that we are building a foundation for students interested in engaging with AI to meet one another, learn real skills, and develop an appreciation for the importance of understanding the global implications of AI in a wide variety of industries and economies.
In only our first year of ratified operation under the University Students’ Council (USC), we have officially engaged over 170 student members and partnered with numerous professors, industry experts, and other student organizations to create a remarkably strong community in an impressively short timespan. We believe there is so much room to grow, with countless opportunities.
Sources:
Analytics Vidhya implementation of CSRNet:
www.analyticsvidhya.com
PyTorch in Flask:
github.com
Original CSRNet Paper: