Real-time and video processing object detection using Tensorflow, OpenCV and Docker. | by Léo Beaucourt | Towards Data Science

In this article, I will present how I managed to use Tensorflow Object-detection API in a Docker container to perform both real-time (webcam) and video post-processing. I used OpenCV with python3 multiprocessing and multi-threading libraries.
I will focus on hurdles I have encountered, and what solutions I have found (or not!). The full code is on my Github.
I started from this excellent Dat Tran article to explore the real-time object detection challenge, leading me to study python multiprocessing library to increase FPS with the Adrian Rosebrock’s website. To go further and in order to enhance portability, I wanted to integrate my project into a Docker container. Main difficulty here was to deal with video stream going into and coming from the container.
In addition, I added a video post-processing feature to my project also using multiprocessing to reduce processing time (which could be very very long when using raw Tensorflow object detection API).
Both real-time and video processing can run with high performances on my personal laptop using only 8GB CPU.
I will not spend time describing Tensorflow object-detection API implementation, since there is ton of articles on this subject. Instead, I will show how I use Docker in my all-day jobs as data scientist. Just note that I used the classical ssd_mobilenet_v2_coco model from Tensorflow for speed performance. I copy the model (the .pb file) and the corresponding label map locally (in the model/ directory) to keep the possibility to use personal model later.
I believe that using Docker today become a primary data scientist skill. In data science and machine learning world, lots of new algorithms, tools and programs are released every weeks and install them on your computer to test them is the best way to crash your OS (experienced!). To prevent this, I now use Docker containers to create my data science workspaces.
You can find on my repository the Dockerfile I’m working with for this project. Here is how I installed Tensorflow object-detection (follow the official installation guide):
# Install tensorFlowRUN pip install -U tensorflow# Install tensorflow models object detectionRUN git clone https://github.com/tensorflow/models /usr/local/lib/python3.5/dist-packages/tensorflow/modelsRUN apt-get install -y protobuf-compiler python-pil python-lxml python-tk#Set TF object detection availableENV PYTHONPATH "$PYTHONPATH:/usr/local/lib/python3.5/dist-packages/tensorflow/models/research:/usr/local/lib/python3.5/dist-packages/tensorflow/models/research/slim"RUN cd /usr/local/lib/python3.5/dist-packages/tensorflow/models/research && protoc object_detection/protos/*.proto --python_out=.
Similarly, I installed OpenCV:
# Install OpenCVRUN git clone https://github.com/opencv/opencv.git /usr/local/src/opencvRUN cd /usr/local/src/opencv/ && mkdir buildRUN cd /usr/local/src/opencv/build && cmake -D CMAKE_INSTALL_TYPE=Release -D CMAKE_INSTALL_PREFIX=/usr/local/ .. && make -j4 && make install
Image building is a bit long and take several minutes. Then, using it is quick and easy.
I first try to apply object detection to my webcam stream. The main part of this work is fully described in the Dat Tran’s article. The difficulty was to send the webcam stream into the docker container and recover the output stream to display it using X11 server.
With Linux, devices are found in the /dev/ directory and can be manipulated as files. Commonly, your laptop webcam is the “0” device. To send its stream into docker container, use the device argument when running the docker image:
docker run --device=/dev/video0
For Mac and Windows users, the way to send the webcam stream into containers is not as simple as for Linux (despite Mac is based on Unix). I have not dug so much into this problem, but the solution for Windows user would be to use Virtual Box to launch the docker container.
Here is the point which takes me some time to resolve (with an unsatisfactory solution). I found useful information on using graphical user interfaces with Docker here, in particular to connect a container to a host’s X server for display.
First, you must expose your xhost so that the container can render to the correct display by reading and writing though the X11 unix socket. Start by setting the permissions of the X server host (this is not the safest way to do it) to let docker access it:
xhost +local:docker
Then, once you are finished using the project, return the access controls at their default value:
xhost -local:docker
Then, create two environment variables XSOCK and XAUTH:
XSOCK=/tmp/.X11-unixXAUTH=/tmp/.docker.xauth
The first refers to the X11 Unix socket, the second refers to an X authentication file with proper permissions we create now:
xauth nlist $DISPLAY | sed -e 's/^..../ffff/' | xauth -f $XAUTH nmerge -
Finally, we just have to update our docker run line command. We forward our DISPLAY environment variable, mount a volume for the X11 Unix socket and the X authentication file with an environment variable called XAUTHORITY which link to it:
docker run -it --rm --device=/dev/video0 -e DISPLAY=$DISPLAY -v $XSOCK:$XSOCK -v $XAUTH:$XAUTH -e XAUTHORITY=$XAUTH
Now we could run our docker container and it’s done:
Despite the host’s X server configuration, I was not able do completely remove what seems to be a bug in my code. OpenCV need to be “initialize” by calling a python script (init-openCV.py) using the cv2.imshow function. I get the following error message:
The program 'frame' received an X Window System error.
Then, it is possible to call the main python script (my-object-detection.py) and the video stream is well sent to the host display. I’m not very satisfied with the solution of using a first python script to initialize the X11 system, but I have not found anything that solves this problem so far.
EDIT: I finally (and accidentally!) found a solution to this problem by using a stable version of OpenCV (3.4.1) instead of cloning the git repo locally. Therefore, there is no need now to call the init-openCV.py before the main python script.
To manage to run the object-detection API in real-time with my webcam, I used the threading and multiprocessing python libraries. A thread is used to read the webcam stream. Frames are put into a queue to be processed by a pool of workers (in which Tensorflow object-detection is running).
For video processing purpose, it is not possible to use threading since all video’s frames are read before workers are able to apply object-detection on first ones put in the input queue. Frames which are read when input queue is full are lost. Maybe using a lot of workers and huge queues may resolve the problem (with a prohibitive computational cost).
Another problem with simple queue is that frames are not published in output queue with the same order as in the input queue, due to ever-changing analysis time.
To add my video processing feature, I remove the thread to read frames. Instead, I used the following lines of codes to read frames:
while True:  # Check input queue is not full  if not input_q.full():     # Read frame and store in input queue     ret, frame = vs.read()      if ret:                    input_q.put((int(vs.get(cv2.CAP_PROP_POS_FRAMES)),frame))
If the input queue is not full, the next frame is read from the video stream and put into the queue. Else, nothing is done while a frame is not getting from the input queue.
To address the problem of frame order, I used a priority queue as a second output queue:
Frames are read and put into the input queue with their corresponding frame numbers (in fact a python list object is put into the queue).Then, workers take frames from the input queue, treat them and put them into the first output queue (still with their relative frame number).
Frames are read and put into the input queue with their corresponding frame numbers (in fact a python list object is put into the queue).
Then, workers take frames from the input queue, treat them and put them into the first output queue (still with their relative frame number).
while True:  frame = input_q.get()frame_rgb = cv2.cvtColor(frame[1], cv2.COLOR_BGR2RGB)  output_q.put((frame[0], detect_objects(frame_rgb, sess, detection_graph)))
3. If output queue is not empty, frames are extracted and put into the priority queue with their corresponding frame number as a priority number. The size of the priority queue is set, arbitrary, to three times the size of the others queues.
# Check output queue is not emptyif not output_q.empty():  # Recover treated frame in output queue and feed priority queue  output_pq.put(output_q.get())
4. Finally, if output priority queue is not empty, the frame with the highest priority (smallest prior number) is taken (this is the standard priority queue working). If the prior corresponds to the expected frame number, the frame is added to the output video stream (and write if needed), else the frame is put back into the priority queue.
# Check output priority queue is not empty  if not output_pq.empty():    prior, output_frame = output_pq.get()    if prior > countWriteFrame:      output_pq.put((prior, output_frame))    else:       countWriteFrame = countWriteFrame + 1          # Do something with your frame
To stop the process, I check that all queues are empty and that all frames have been extracted from the video stream:
if((not ret) & input_q.empty() &     output_q.empty() & output_pq.empty()):  break
In this article, I present how I used docker to implement a real-time object-detection project with Tensorflow. As said, docker is the safest way to test new data science tools as well as to package the solution we deliver to customers. I also show you how I have adapted the original python script from Dat Tran to perform video processing with multiprocessing.
Thanks you if you read this article from the beginning to end! As you have seen, there are lots of possible improvement with this project. Don’t hesitate to give me some feedback, I’m always keen to get advices or comments.