A Note on OpenCV Threads & Performance in Prod.
By: Rachit Tayal
Tags: Computer Vision, Prod, Opencv, Docker

A Note on OpenCV Threads & Performance in Prod.

Disclaimer: OpenCV can be the real villain and can severely impact the performance of your computer vision application esp. in production having kubernetes environment (using dockers). Rachit Tayal May 14, 2019·5 min read

Motivation

I have been working on Computer Vision for sometime now which involves building applications involving image processing, deep learning and productising them at the same time. Recently I stumped upon an issue in my application (will abbreviate as app from here) which was degrading the performance of it in the deployment environment. The issue was caused by the default behaviour of the OpenCV library (provides functions/methods for various image processing tasks such as reading frames, resizing etc.) which is used extensively in vision based application, but not being aware of it at that time, I ended up wasting hours to debug it, only adding to my frustration and caffeine intake ;).

Nevertheless Since I was unable to find a proper blogpost on it and would not want any developer doing the same, I’ve decided to write a small post to summarise my observations that might come in handy.

Therefore, the aim of this post is to make developers aware of a small yet important issue that can hinder the performance of any vision based app significantly. Especially if dockers are used for deployment, there is a high chance that developers may stumble across the same scenario. Nevertheless, I believe developers must be aware of the behaviour discussed below, therefore it’s worth a read.

The Odd Behaviour

For any computer vision application, usually the flow involves reading frames from some source (Video file, Camera stream etc.) doing some processing on the frames, which in my case was performing object detection using deep learning and getting the desired output. OpenCV library methods are extensively used in above flow from reading frames to resizing, visualising them etc.

The observation I made was, that the time taken to process one frame in production env. was 3x times more as opposed to what it is taking when run on a local machine. To sum it up:

Summary

Weird, right?? Let’s dig deeper to understand as to why that is happening!

On the side note, in case you are wondering, how I assigned 8 cores to my app. (out of 24), for that our production env. is k8 compliant, hence we wrap app in docker image and deploy. Therefore I specify the -cpus option during the docker run command execution. If you are not familiar with dockers, you can refer to this blogpost to develop a basic understanding.

The Core Problem & its consequences

At first It seemed to be an inference engine issue (used for object detection) but as I dig deeper I observed, the threads spawned by the app in prod. are way more than the threads spawned on local machine.

Summary

Due to large number of threads being spawned, unnecessary overhead will be created since all the threads will be competing for CPU hence the cost of context switching will be significant, adding to the processing time of frame. Plus the overhead caused by the inter-thread communication will also make matters worse.

Therefore, the performance of the app takes the hit and we see the higher processing time in prod. Also the load average on the node (on which the app is running) will increase drastically, since many threads are waiting in the task queue which are not even required at the first place, therefore it will also be affecting the performance of other apps running on the same node.

Guess who is the culprit for spawning multiple threads — yes, OpenCV, but why?

If you use OpenCV library beware that it spawns multiple threads for image processing internally. For example, cv.VideoCapture() spawns multiple threads internally.

The number of threads spawned depend on several factors including the underlying hardware spec, method invoked etc, but generally speaking, by default, OpenCV reads the underlying hardware spec. and spawns that many number of threads as the cores present in the machine. If we see the math of threads spawned in above table:

Locally: Total Threads spawned = 13

13 = 8 (created by OpenCV, since 8 cores) + 5 (other module of app) In Prod: Total Threads spawned = 29

29 = 24 (created by OpenCV, since 24 cores) + 5 (other module of app)

General rule of thumb is, your app should be spawning that many threads as the cores it has available for processing. Therefore no concerns are raised when the app is run locally since number of cores available are equivalent with threads spawned by OpenCV, hence no impact on the performance. But assigning 8 cores to our docker container running the app, OpenCV will be spawning extra threads in prod. since the node on which it is running has higher number of cores available, which will start hindering the performance.

Solution

Well, you guessed it right. Explicitly informing the OpenCV on how many threads to spawn should solve the problem. You can achieve this via cv::setNumThreads(int) if you’re using the C++ library, or cv2::setNumThreads(int) , if you are using the Python wrapper.

Note: Generally setting it to the number of cores available for processing is optimal, however you may have to test it with multiple values for even optimal performance. Moreover I feel it’s not the fix but the issue which is usually not visible to the developer as everything seems to be working seamless unless deployed in prod :-)

Conclusion

If you use OpenCV library beware that it spawns multiple threads for image processing internally. Therefore always explicitly set the number of threads to be spawned by OpenCV based on how many cores your app is going to use in prod, else performance of the app can take a drastic hit.