Realtime video object detection on Raspberry Pi | by Constantin Toporov | Towards Data Science

Recently a video from Tesla autopilot view appeared on the Internet and got very popular. I was interested in the streaming of a similar video (with the detected objects) for a long time and this demo finally made me do some work.
The only problem is I wanted to stream video from Raspberry Pi, but in general, it is not powerful enough to perform such tasks.
The most straightforward implementation is to run a detector on Raspberry Pi via OpenCV-DNN.
OpenCV-DNN supports multiple networks and formats, but I used to work with MobileSSD from Google (version 11_06_2017, the latest one is not compatible with OpenCV 4.2).
First, read classes names and load the model:
tf_labels.initLabels(dnn_conf.DNN_LABELS_PATH)net = cv.dnn.readNetFromTensorflow(dnn_conf.DNN_PATH,         dnn_conf.DNN_TXT_PATH)
Then run inference on an image:def inference(img): net.setInput(cv.dnn.blobFromImage(img, 1.0/127.5, (300, 300), (127.5, 127.5, 127.5), swapRB=True, crop=False)) return net.forward()
This operation takes 1.7 seconds on Raspberry. For example, my pretty old laptop does the same in 0.2 sec.
Transform the output tensor into human-friendly JSON:
def build_detection(data, thr, rows, cols):        ret = []         for detection in data[0,0,:,:]:                 score = float(detection[2])                 if score > thr:                         cls = int(detection[1])                         a = {"class" : cls, "name" : tf_labels.getLabel(cls),  "score" : score}                         a["x"] = int(detection[3] * cols)                         a["y"] = int(detection[4] * rows)            a["w"] = int(detection[5] * cols ) - a["x"]                        a["h"] = int(detection[6] * rows) - a["y"]                         ret.append(a)         return ret
Considering the measurements, it is obvious the native Raspberry performance (1.7 sec) is not enough for any video.
But making Raspberry cooperating with the laptop should increase the total ability of handling frames.
Now wrap the detector by a REST interface.
I usually use Flask for this purpose. Got service with two entry points:
the first returns detections in JSON
the second returns an image with detected rectangles and labels
The second method gives a picture of detections as easy as:
curl -X POST -F "file=@pic.jpg" host:80/ddetect --output a.jpg
We want a scalable and portable solution then pack the service into a docker image.
Picked Debian Stretch + Python as the base image to remain with a similar stack as Raspbian.
Need to set up flask, protobuf, requests, opencv_python, download Mobile SSD and the server code from GitHub.
FROM python:3.7-stretchRUN pip3 install flaskRUN pip3 install protobufRUN pip3 install requestsRUN pip3 install opencv_pythonADD http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v1_coco_11_06_2017.tar.gz /RUN tar -xvf /ssd_mobilenet_v1_coco_11_06_2017.tar.gzADD https://github.com/tprlab/docker-detect/archive/master.zip /RUN unzip /master.zipEXPOSE 80CMD ["python3", "/docker-detect-master/detect-app/app.py"]
In order to use the image with third-party providers, one needed to publish it on some public docker registry.
I decided to go the traditional way with Docker Hub.
Create an account
Authorize local docker with DockerHub: docker login
Name the image: docker tag <local_image_name> tprlab/opencv-detect-ssd
Upload the image: docker push tprlab/opencv-detect-ssd
Now the image can be used with many container providers like Google Cloud, Microsoft Azure, Amazon Web Services, Digital Ocean, Heroku, etc.
Using an OpenCV example, make a basic server:
def handle_frame(frame):    return detect.detect_draw_img(frame)       def generate():    while True:        rc, frame = vs.read()        outFrame = handle_frame(frame)        if outFrame is None:            (rc, outFrame) = cv.imencode(".jpg", frame)        yield(b'--frame\r\n' b'Content-Type: image/jpeg\r\n\r\n' + bytearray(outFrame) + b'\r\n')@app.route("/stream")def video_feed():    return Response(generate(), mimetype = "multipart/x-mixed-replace; boundary=frame")
With the docker deployed on the laptop, the video goes in slow motion.
So far, a single docker is not enough. Let its scale — in this case, we will be able to handle multiple frames simultaneously.
I selected GCloud as a service provider for Kubernetes and run my image in 5 instances.
The server code became more compilated:
def generate():    while True:        rc, frame = vs.read()        if frame is not None:            future = executor.submit(handle_frame, (frame.copy()))            Q.append(future)keep_polling = len(Q) > 0        while(keep_polling):                        top = Q[0]            if top.done():                outFrame = top.result()                Q.popleft()                if outFrame:                    print("Frame", datetime.datetime.now())                    yield(b'--frame\r\n' b'Content-Type: image/jpeg\r\n\r\n' + bytearray(outFrame) + b'\r\n')                keep_polling = len(Q) > 0            else:                keep_polling = len(Q) >= M
This approach helped to stream a detected video with about 10 fps speed.
And there is a space to increase the speed — just add more nodes in our Kubernetes cluster.
After experiments with remote detection, I decided to try a local hardware solution.
I have an Intel NCS and already used it in similar tasks.
So I converted MobileSSD into OpenVino format and run the video streamer with the NCS detector.
NCS does this job pretty good — FPS was slightly over 10, outperforming 5-nodes Kubernetes cluster.
Eventually, it is turned out to be possible to run near-real-time detections using external power.
Using NCS is the easiest approach (and cheapest) — buy once, worked for a long time with the simplest code.
The docker way is more sophisticated but also more scalable, portable and reusable.
The GitHub repo with Dockerfile and source codes
Ready-to-go docker image on Docker Hub