openCV GPU usage disappoints.... Use the GPU with your openCV code, get... | by Peter Gaston | Towards Data Science

Several times I’ve been drawn to the siren call of using key openCV calls on the GPU. Most recently for the matchTemplate call — before that for calls relating to visual odometry. Ah, disappointment!
The situation now is that I have 31 possible images (or templates) to match against the image from the sensor. I want to find out which one fits best — anywhere on the big image. Using the openCV call matchTemplate for each of the 31 images, I get a score for the best match — the highest score is the winner. I plan on running this on an nVidia single board computer — whether nano, Xavier or whatever is appropriate for cost and performance.
First, the basics of the code being used (boring parts omitted...)
// Initialize the GPU TemplateMatching object// - running in 8 bit modecv::Ptr<cv::cuda::TemplateMatching> cvCuda;cvCuda = cv::cuda::createTemplateMatching(CV_8U, cv::TM_CCORR);// , Size(0, 0));// Initialize the big image - in GPU landcv::cuda::GpuMat img_Gpu;img_Gpu.upload(inputImg);// Initialize the templates for matching - in GPU landfor (int i=0; i<kTemplateAngles; i++)  {    cv::cuda::GpuMat tmplGMA_GPU;    tmplGMA_GPU.upload(newGMA);    vTmplGMA_Gpu.push_back( tmplGMA_GPU );}// Initialize the results matrix in GPU landint result_cols =  monoImg.cols - baseGMAImg.cols + 1;int result_rows = monoImg.rows - baseGMAImg.rows + 1;cv::Mat result( result_rows, result_cols, CV_32FC1 );cv::cuda::GpuMat result_Gpu( result_rows, result_cols, CV_32FC1 );// start timer - and do more than one loop to take multiple seconds// loop through our 31 templatesfor (int i=0; i<kTemplateAngles; i++)  {    cvCuda->match( img_Gpu, vTmplGMA_Gpu[i], result_Gpu );    cv::cuda::minMaxLoc( result_Gpu, &minVal, &maxVal, &minLoc, &maxLoc );}// end timer
Fine — so let’s jump to the results:
On the desktop, the GPU was nearly 3x slower! On the Nano — only almost 2x slower.
This seems to be a common complaint on the internet. The main issues outlined seem to be:
Initialization (of context) can take up to 10 seconds apparently. I’ve done the timed looping multiple times — post initialization. The second time is marginally faster, say 15% but subsequent to that the times are stable (and are what are shown in the table above.)
The movement of data to the GPU memory takes time. Certainly, which is why I tried to do this all ahead of time.
Perhaps others, feel free to comment in at the bottom. And...
The actual execution time of the GPU is relatively small, compared to the overhead of CUDA in getting it started/ended — that any potential gains of running on the fast GPU are lost.
The last, I think is the core problem. That is, I’m not doing enough on the GPU to justify all overhead incurred by the calls back and forth. (Now if only there were an openCV wrapper on matchTemplate — say to iterate through N templates...)
By the way, there are opportunities where it still could make sense to use the GPU, even if it takes more time than the CPU. In the first case where I did this investigation, we were CPU bound. Offloading anything to the GPU would free up the CPU to perform other tasks — as long as we didn’t take too long!
In summary, perhaps I’m missing something, but twice I’ve struck out on hoping for speed improvements using the GPU on openCV calls.