Video analytics at the edge. Use Raspberry PI and Intel NCS to... | by Aby Varghese | Towards Data Science

I have recently installed a CCTV system at my home. It consisted of 4 IP cameras. I wanted them to be a bit more smarter. Instead of looking at the CCTV footage always, it would be good to get a notification if there is human detected by any of the cameras. This is a really cool stuff. I thought it would be really good if I get an alarm in my Telegram if any of the camera detect a motion. Though I started it for fun, I had a great learning. Fundamentally it enables video analytics at the edge without needing a cloud. I am here to share my learning and eager to learn from you if there are ways to optimize.
I have 4 different IP cameras that stream 720P videos at 25FPS. I have a raspberry pi sitting idle and have got an Intel Movidius neural compute stick (ie first generation of NCS ). I wanted the Raspberry PI and NCS to perform the human detection and send me the notification in telegram when the camera detects a person. There were few things that I consider very important during the build, which are
1. I should be able to run the application almost 24/7 , that means the code should be efficient. The CPU utilization must be as minimum as possible so that the PI’s temperature is under control.
2. I should get the notification almost instantaneously. The highest priority was that the human detection should complete as soon as it happens.
3. The notification should contain an image so that I can understand the context of the alert (who was the person).
Let me tell you in advance, the item 1, and 2 were much harder than I thought.
I thought it is an easy task assuming Raspberry pi can get the video frame from each of the camera and NCS can perform the human detection. However, it was not so easy.
As usual, I started with plain OpenCV code to read the video RTSP stream a single camera and see how it works.
cap = cv2.VideoCapture(“RTSP URL of the IP Camera”) while(True):    ret, frame = cap.read()    cv2.imshow('frame',frame)    if cv2.waitKey(1) & 0xFF == ord('q'):        break cap.release()cv2.destroyAllWindows()
However few things were not working as I expected. First and foremost, there were huge delay in getting the frame in real time. Ie even if there are no processing (human detection) there was a delay of around 5 to 10 seconds and in some cases, it went to 40 seconds within couple of minutes of running the application. The CPU utilization reached 100% immediately. This is the point where I realized this is much complex than I thought. So, I had to go back to the drawing board to see what would be the best way to solve this problem.
I observed that this solution does not require to process each and every frame from the camera during my analysis. We absolutely don’t need to analyze 25 frames that are generated in a second. This is one of the primary principles that I used in my design and I came up the following design. What parameter will help us to define the number of frames that the system can process will be discussed later.
As you may notice, the idea is to capture N number of frames from each of the camera in every X seconds. This was not the only reason which made the system work efficiently. The solution made use of the hardware acceleration available in Raspberry pi so that the CPU utilization is very minimal. This will be detailed when I describe about GStreamer. The reason why the number of frames and the duration are variable is that it is purely based on the processing power of the hardware that is used for processing the frames. As per my knowledge, the first version of Intel NCS can perform less number of inferences. As you can imagine, the diagram clearly shows the bottle neck in the processing engine since it has to process all the frames. Since my plan to make use of an existing Intel NCS, I had to adjust these parameters according to the capabilities of the NCS.
As a starting point, I decided to capture one frame in every one second from each of the camera. This seems to be reasonable to me. Since it is not practically possible for a person to go past the camera viewing area in less than a second. However there is a possibility that the processing engine might fail to detect the object, but that is a separate topic which we will look later.
I realized that it is difficult to specify the frame number(s) / second that I want to capture when I am using OpenCV. So explored and found out that there are possibly two approaches to solve this problem . One is GStreamer and another is FFMPG. So I decided to explore the first option GStreamer further.
I found GStreamer is extremely powerful but complex due to the lack of documentation / examples. So I started exploring few tutorials in youtube and I found them very useful. I have mentioned the links that helped me to study about GStreamer in the Reference section. I am sure it will help you too.
The source code for this experiment can be found here :
github.com
The first step for me was to install GStreamer in Raspberry PI. This was fairly straight forward.
sudo apt-get install gstreamer1.0-tools
The next step was to install Intel OpenVino toolkit for raspberry PI. This was necessary since the processing engine was Intel NCS. I had to install a version of OpenVINO that is supporting intel NCS since the latest versions of OpenVINO only support the Intel NCS 2. Please follow the link below link and install OpenVINO based on the NCS that you have.
docs.openvinotoolkit.org
Other libraries include:
pip install telepotpip install viztracersudo apt-get install pkg-config libcairo2-dev gcc python3-dev libgirepository1.0-devsudo apt-get install python3-gipip install gobject PyGObject
The next step is to create the GStreamer pipeline. As I mentioned earlier, it is very powerful tool. Gstreamer is configured by the use of pipelines, which are a series of commands specifying where to get video from, how to process and encode it, and then where to send it out.
Lets have a look at the pipeline that I have created.
pipelinestring = "rtspsrc location={0} name=m_rtspsrc ! rtph264depay ! h264parse ! v4l2h264dec capture-io-mode=4 ! video/x-raw ! v4l2convert output-io-mode=5 capture-io-mode=4    ! video/x-raw, format=(string)BGR, width=(int)640, height=(int)480 ! videorate ! video/x-raw,framerate=1/1 ! appsink name=m_appsink sync=false"
Lets see the purpose of each element.
rtspsrc location={0} — The location is the RTSP url of the camera stream.
rtph264depay — Extracts H264 video from RTP packets.
v4l2h264dec- This is an interesting one . This tells the Raspberry PI to decode the H264 using the GPU. Remember there is an alternative which uses the CPU to decode. It is avdec_h264. Using avdec_h264 might work if we try to decode from only one camera. However since our objective is to capture from 4 cameras, it is very essential for us to offload it to the GPU.
Lets look into the parameters,
Offloading the video decoding to GPU is a critical step towards our goal of minimizing the usage of CPU.
v4l2convert — This is letting the system know to convert the frame using the GPU instead of the CPU. There is a CPU equivalent which is video convert. This is again a CPU intensive process so we should be using v4l2convert to make use of the hardware acceleration offered by the Raspberry PI. We are doing few conversions here, they are
converting the source format into BGR. The frame size is reduced to 640x480 (from the original (720P). The documentation for v4l2convert says that there is an option for controlling the frame rate using v4l2convert itself. However I couldn’t make it work. Please let me know if any of you have made it work, I would be glad to hear. I had to use another videorate to control the frame rate since I couldn’t make v4l2convert control the frame rate.
videorate — This will help us to control the framerate that we want to capture in the application. Here in our case we want to capture only 1 frame in a second from a camera. So it will be framerate=”1/1” . Assume we want to capture only one frame in two seconds then the framerate will be “1/2”. Incase if we want to capture 5 frames in a second, the frame rate will be “5/1”
Now the idea is to run the video capture application using multiprocessing to make use of the different cores of Raspberry PI. The main program will kick off a separate process for each of the camera.
The rest of things are fairly straight forward. Once the frame is captured, it is converted to OpenCV format and pushed into a Queue. Currently this conversion is done using python code, kindly let me know if there is way to handle it within GStreamer pipeline. I am assuming there will be, but I have not investigated. It can further reduce the CPU usage if we can handle it within GStreamer pipeline as per my knowledge. Since we have 4 different cameras, there will be 4 different queues. The queues are using TCP protocol, because it will help us to distribute the load in future if we want this system to work on multiple hardware. ie we can scale out the processing server based on our needs and increase the frame rate or interval of capture or add more cameras. Probably we shall optimize it for distributed environment in the next release.
In the main_prgall.py (line number 27), you will have enter the RTSP URL of your streams
self.camlink1 = 'rtsp://<username>:<password>@192.168.1.2:554/cam/realmonitor?channel=1&subtype=0&unicast=true&proto=Onvif' #Add your RTSP cam link
and add the rest of the stream URLs based on your environment.
You can consume GStreamer pipeline from OpenCV. The Opencv needs to build with GStreamer support.
Processing Server
The idea is very straight forward, grab the frame from the Queues (4 queues) and push it to the intel NCS. The queue works using PUB / SUB model here. The camera is the PUB and the process server is the SUB. We shall move a frame to another queue if NCS detects a human in the frame.I have used the person detection retail 013 model from the intel model zoom. The details of the model can be found here:
docs.openvinotoolkit.org
It is essential that you should use the model that is compactible with your device. If you have NCS 2, then you should use the model that support NCS2.This is a model that can detect person in the video frame. It accepts the input frame of size height 320 and width 544
Communication engine
This is a very straight forward component. It gets the processed data from the Queue and stores the image into the local directory. It also connects with the Telegram bot and sends out a notification with the image attached.
The Bot Key and user ID is removed from the source code. You will have to insert your own Telegram bot Key and user ID
You can see from the image that the frames are processed almost realtime. During my testing, I found that all the frames were able to process in less than 2 seconds. ( detect motion in less than 2 seconds ). There might be a slight delay in the Telegram bot sending the image, but that is based on your Internet connectivity . However, I found it was negligible in almost all the cases.
CPU Consumption
The CPU consumption was around 30–35% which is okay considering the fact that it is processing the videos from 4 different streams in every second. The below image shows the CPU consumption and the temperature reported by raspberry pi.
I have found another way in which CPU utilization can be reduced. That is not in the python scripts, but in the CCTV itself. Most of the IP cameras supports multiple streams. ie one Mainstream ( with High quality resolution and frame rate) and sub stream with reduced resolution(640x480 in my case). The following screenshot shows the setting in my camera web interface
The following diagram shows the CPU utilization and temperature when connected to sub stream
As you may notice , the overall cpu utilization is between 16–22% that seems to be very reasonable. Since the heavy lifting was done the GPU, the system is able process multiple streams with less CPU usage in less than 2 seconds
Raspberry PI is a great platform. You will be able to find plenty of libraries and support .It is relatively cheap and efficient. The Intel NCS 1 does its duty. Probably the Intel NCS2 will be able to perform more number of inferences because it is advertised as much better in terms of performance. I have not got a chance to test the application in NCS 2 since I dont have one. I would really love to hear If any of you would like to test the script in NCS2 and report the performance . Having said, it just happened that I had both NCS1 and Raspberry PI for my experimenting. This might not be the best platforms if you are starting from scratch from a cost perspective. The RPI will cost around 4K INR and NCS costed me around 8K INR. I am curious to know how Nvidia’s Jetson Nano 2GB is. It has much better performance than NCS2 and dont require another SBC like PI. The cost is around 5K INR. Probably I shall cover it in my next update.
References
https://github.com/sahilparekh/GStreamer-Python — This is great repo that shows how to fetech RTSP streams in OpenCV. I have extended this sample in my project and the credit goes to the respective owner.
https://www.youtube.com/watch?v=HDY8pf-b1nA — This is a great place to start learning about GStreamer . This has really helped me to learn about GStreamer.
http://lifestyletransfer.com/how-to-use-gstreamer-appsrc-in-python/ — This is great place to learn about advanced Gstreamer concepts . A great set of working examples with great explanations.