Building a frontend for a Raspberry Pi Machine Learning Server with Intel Movidius NCS 🚀
By: Alexey Györi
Tags: Raspberry Pi, Openvino, Computer Vision, Edge Computing, Christmas

Building a frontend for a Raspberry Pi Machine Learning Server with Intel Movidius NCS

System diagram visualizing the flow of information via numbers. Click here for the draw.io diagram. Diagram by author.

This is part 2 of a series on Computer Vision (CV) applications for the RaspberryPi and the Intel Movidius Neural Compute Stick 2 (NCS)¹. Check out part 1 on running object recognition with OpenVino.

Outline

· QUICKSTART / TL;DR

· Motivation

· Requirements

· 0. Overview

· 1. Machine Learning Model

· 2 Setting up a Flask Backend and Webserver

· 3. Writing the JavaScript frontend

· 4. Starting the server

· 5. Results

· Footnotes

QUICKSTART / TL;DR

Set up OpenVino2020_4² with an Intel Neural Compute Stick 2 on your raspberry pi and clone this Github repository onto it.

Verify that everything is set up correctly via

./test

Start the application via:

./start

You can now navigate to your pi via typing https://raspberrypi.local:5000³ into a browser. The application is configured to detect pedestrians from the back camera of your phone or laptop camera.

Motivation

Intel’s OpenVino toolkit makes it easy to deploy ML models from a large model zoo. It even includes a python demo for object classification which uses asynchronous processing. However, when using just the given pure-python demo for any video processing, you need to

A. physically attach the camera to the pi

B. do the video feed rendering on the pi which offloads additional load onto its limited CPU.

Instead, to offset these shortcomings, we run the model inside a flask server and offload any video capturing and rendering operation onto the client. This makes real-time high framerate applications possible with relatively low energy consumption. Also, the Pi can act as a kind of persistent storage for all of your detections.

One example application of this could be to create a security camera from an old Android phone.

Requirements

Python3 with OpenCV. This comes with the Intel Movidius installation for Raspbian.

A Raspberry Pi. I am using a Raspberry Pi 4 Model B. However, since any compute-heavy operations are offloaded onto the client and the compute stick, the model will not matter.

An Intel MOVIDIUS NCS2 stick. [see 1]

0. Overview

Its possibility of direct DOM manipulation, great speed, and asynchronous nature makes JavaScript a great partner for Python-powered Data Science applications. Official Python Logo | Unofficial JavaScript Logo from Wikimedia | Iceberg from @anniespratt | Combined gif by author

1. Machine Learning Model

1.1 Setting up our model

We will be using a pre-trained neural network from the OpenVino model zoo which will detect whether there is a pedestrian or not in the current camera feed.

An example detection by the pedestrian detection model. Picture from Mary Taylor @Pexels

An example picture of the person detection model. Picture courtesy of Intel, taken from https://docs.openvinotoolkit.org/2018_R5/_docs_Retail_object_detection_pedestrian_rmnet_ssd_0013_caffe_desc_person_detection_retail_0013.html

It is important to note that models are OpenVino version dependent. I.e., you cannot run a 2019_R1.1 model on OpenVino 2021.1 (see ²). Hence, the link below will be different depending on which version of OpenVino you installed. To download other versions go to the root https://download.01.org/opencv/ dir, select your version of the toolkit and go to open_model_zoo/. Use the FP16 version⁴

# create a folder that will contain the model

mkdir model && cd model # -- download models for 2020_R4--

wget # get the xml file describing the networkwget https://github.com/AlexeyGy/raspberry-frontend/raw/master/model/person-detection-retail-0013.xml

wget # get the network weightswget https://github.com/AlexeyGy/raspberry-frontend/raw/master/model/person-detection-retail-0013.bin cd ..

create a new file called model.py which is going to do the inference.

The set_up_inference() function that we just wrote will initialize our neural network for further use and return a net object . Note that OpenVino ships with a modified version of OpenCV which contains additional APIs for working with OpenVino models.

Let’s add a recognize method which will perform the inference⁶ and will return the bounding-box coordinates of detections.

Interesting lines

in line 27, the image is resized to fit the expected size by the network. ddepth here means the number of channels of each individual pixel

line 33 reshapes the network output into a list of [Detection1{image_id, label (not used for this model), confidence, x_min , y_min, x_max, y_max }, Detection2{same as detection1},…] This format is given by the model

, }, Detection2{same as detection1},…] This format is given by the model line 40 will filter detections that are less than the confidence threshold.

You can download the full model.py file here

1.2 Testing our model

To test that the detection produces the expected results in order to sanity check that everything is set up correctly, let’s write a test⁷.

Firstly, let’s add two images of test data. One with just a person’s face and one with a full body.



wget

wget

cd .. mkdir -p test-images && cd test-imageswget https://github.com/AlexeyGy/raspberry-frontend/raw/master/test-images/face-exists.jpg wget https://github.com/AlexeyGy/raspberry-frontend/raw/master/test-images/person-exists.jpg cd ..

Since we are using the pedestrian detection model, we would expect that the face is NOT recognized while the person standing on the mountain IS recognized. Create a file test_detecting.py

You can run the test via executing python3 -m unittest .

if you get an error that cv2 does not exist => check if your OpenVino environment is initialized

if you get weird errors during recognize, => check if you have the right model version for your inference engine

This concludes the work on our model 🎉.

2 Setting up a Flask Backend and Webserver

create a file named app.py.

Here, we are going to 1. write a process function which will process images sent to it via the /upload route:

There are three routes defined here:

/upload which accepts images sent to it by javascript from the frontend file.

=> we are using our model from model.py to set up our neural net

=> _save_detection will add bounding boxes to the image (for input formatting check model.py)and save the result in the defined output folder with a name like 2020–12–01.jpg

which accepts images sent to it by javascript from the frontend file. => we are using our model from model.py to set up our neural net => _save_detection will add bounding boxes to the image (for input formatting check model.py)and save the result in the defined output folder with a name like 2020–12–01.jpg / which will statically serve index.html (see 3.1)

which will statically serve index.html (see 3.1) /images which will simply serve all image snapshots that our application takes if they have at least one detection (see 3.2)

We are done with the backend 💖 , for now; starting the server will be handled in 4.

3. Writing the JavaScript frontend

3.1 index.html

Create a static folder and an images output folder, cd into it and download webcam.js. Webcam.js⁸ is a helper library for capturing the webcam feed.



cd static

wget

wget mkdir -p static/imagescd staticwget https://github.com/jhuckaby/webcamjs/raw/master/webcam.swf wget https://github.com/jhuckaby/webcamjs/raw/master/webcam.js

now let’s write index.html and firstly create a skeleton which will contain our detecting function:

the notable lines are

line 12 gets the user to the snapshots folder (see 3.2)

in lines 14–19, we are creating a canvas object which will be used by webcam.js. It is important that the output size matches line 20. Here, you could actually go with a higher output size than 480p depending on your use-case.

line 24 sets the facing mode to environment, this makes sure that on smartphones, we will be using the rear camera by default

line 26 will set a waiting time of 5000ms (5 seconds) before executing main. This is in order to give some time for the webcam to attach.⁹

line 28 defines the main function which will execute take_snapshot every 500 milliseconds (half a second). You may want to set this to be faster for your use-case. The Movidius chip can easily do more.

let’s now look into our main snapshotting function. Add the following code right below line 32:

the take_snapshot function does the snapshot taking and posting the picture to the backend. If the backend responds successfully (code 200)

the predictions are parsed, clearCanvas() removes old bounding boxes drawRect() draws new bounding boxes on the screen. You can change lineWidth and change the bounding box color via ctx.strokeStyle()

Since this function is executed asynchronously to the video rendering on your screen, any video feed being processed can run at the maximum possible fps of the camera (e.g., 30 or 60) and the video feed appears smooth with no lag.

The UI of the system. No detection has taken place since the system reacts to pedestrians. Image by author.

3.2 serve all images from the output folder

Additionally, we would like to be able to see all images that had a detection in a collection. Create the file templates/images.html:

Note that the for loop here is NOT JavaScript. Instead, we are using Flask’s Jinja2 templating engine to pass all filenames in the folder static/images via os.listdir() (see the index.html on top for the code). Flask will dynamically generate HTML code based on the input to display all images¹⁰ that will look like this:

<h1>image1-filename<h1>

<img src="images/image1-filename" alt="image1-filename">

<h1>image2-filename<h1>

<img src="images/image2-filename" alt="image2-filename">

...

4. Starting the server

You can start the server with the following bash script:

More details on the options:

— without-threads makes sure that the flask server does not start multiple threads for processing. This is not strictly necessary but makes sure that we do not

— host: 0.0.0.0 exposes the server to the world

— FLASK_APP sets the file to run as main.py

— FLASK_ENV=development runs flask in

— cert ad-hoc gives us an https certificate. Https is necessary since modern browser enforce it for accessing webcam feeds. Unfortunately there will be an ugly warning in your browser⁵ which you can safely ignore (click on I trust the risks... )¹¹

That’s it, we are done 😎

5. Results

Open https://raspberrypi.com:5000 and ignore the https warning. The model will start detecting after five seconds and will save all images with detections. Open the detections via clicking on the button on the top left.

The resulting service is able to show you the webcam feed fluently at 30fps on any device while doing bounding-box predictions on the pi.

With our system, we are safe from any unexpected visitors on Christmas. Video from cottonbro on Pexels.

And we can prove that this suspicious person really was here. Video from cottonbro on Pexels.

There are a lot of places you could go now. Some ideas:

train your own network for a custom use case like detecting birds, cats or dogs

optimize storage and presentation of the images shown

squeeze the most out of the pi via using a job queue in the model

expose your Pi to the outside world via port forwarding

build a telegram bot that would send you any detections as telegram messages

Did you like this article and want to try something similar on your Christmas break? Post it in the comments. I’d love to hear about other projects in this space.