Object detection model deployment on CPU? Real-life solutions

username: bonzogondo
cooked: <p>Hi,</p>
<p>I am currently working on a small toy-project that involves object detection as one of the steps.<br>
Currently, I amusing a pre-trained Faster-RCNN from <a href="https://github.com/facebookresearch/detectron2" rel="nofollow noopener">Detectron2</a> with ResNet-101 backbone.</p>
<p>I wanted to make an MVP and show it to my colleagues, so I thought of deploying my model on a CPU machine. Detectron2 can be easily converted to Caffe2 (<a href="https://detectron2.readthedocs.io/tutorials/deployment.html#use-the-model-in-c-python" rel="nofollow noopener">DOCS</a>) for the deployment.</p>
<p>I measured the inference times for GPU and the CPU mode. The inference time of the original Detectron2 model using PyTorch and GPU is around 90ms on my RTX2080 Ti.<br>
The converted model on CPU (i9 9940X) and using Caffe2 API took 2.4s. I read that the Caffe2 is optimized for CPU inference, so I am quite surprised by the inference time on CPU. I asked about this situation on the Detectron2 GitHub and I got an answer like: „Expected inference time of R-50-FPN Faster R-CNN on a 8 core CPU is around 1.9s. Usually, ResNets are not used on CPUs.”</p>
<p>There is my question, how such deep learning solutions for Computer Vision should be deployed in the real-world? I read somewhere that Facebook does use Caffe2 for their production models as CPUs are super cheap compared to GPUs (of course they are), but the difference in the running time is really huge. Using CPU for object detection seems useless for any real-time application.</p>
<p>Should I use some other architecture, which does not include ResNet or Faster-RCNN (like YOLO v4/v3, SSD, etc.)? Or maybe the original GPU-trained model should be converted to ONNX and then used in other more CPU-optimized frameworks such as OpenVINO? Or there are some other tweaks such as quantization, pruning, etc. that are necessary to boost the CPU-inference efficiency of production models?</p>
<p>I know that this is just a toy-project (for now at least), I can use GPU for inference (quite costly in real applications?) or just use other architecture (but sacrifice performance). I am just wondering what is the go-to solution for real-world systems.</p>
<p>Thanks in advance for sharing your knowledge and experience. I will be grateful for any hints!</p>

username: WaterKnight
cooked: <p>I am not a big expert in this area but just for letting know what I have discovered.</p>
<p>Converting  PyTorch models to TorchScripts models is a good idea. You get a file that contains the arch class and weights. In the first run of the model gets optimized and after that is pretty fast. I am getting around <strong>1.98 s ± 42.1 ms</strong> in my Ryzen 1700 with Mask-RCNN Resnet 50 (the implementation of torchvision).</p>
<p>I have also discovered <a href="https://pytorch.org/docs/stable/quantization.html" rel="noopener nofollow ugc">quantization</a> . It optimizes models and make them able to run on a mobile or arm devices like Raspberry. However, after quantization the model losses some precision. Here you have an image explaining the process:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/0/203846267f409c47172ee5a4228624792e7ac660.png" data-download-href="https://discuss.pytorch.org/uploads/default/203846267f409c47172ee5a4228624792e7ac660" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/0/203846267f409c47172ee5a4228624792e7ac660_2_443x500.png" alt="image" data-base62-sha1="4B1Qqtn7SJ1oTbmL7lN6IeCRhWo" width="443" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/0/203846267f409c47172ee5a4228624792e7ac660_2_443x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/0/203846267f409c47172ee5a4228624792e7ac660_2_664x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/0/203846267f409c47172ee5a4228624792e7ac660_2_886x1000.png 2x" data-dominant-color="343433"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1396×1574 113 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>You could run this models on your computer not just mobile devices</p>

username: bonzogondo
cooked: <p>Thanks for sharing this.<br>
I am about to check the quantization, but I did not think about using TorchScript. Thanks!</p>
<p>Even though your running time of Mask-RCNN is better than my detector I feel still dissatisfied with inference time around 2s. I would be glad if the detector could run in around 1s; that would be acceptable for my settings, but I am not sure if this is feasible for Faster-RCNN &amp; CPU.</p>

username: FabianSchuetze
cooked: <p>I tried something similar as you did and also began with a RCNN and also had to realize that the RCNN is  indeed slow. In my experience SSD and in particular SSDLite is accurate and fast.</p>

username: bonzogondo
cooked: <p>Yes, I think the RCNN is a dead-end for CPU inference. However, Facebook does use object detection algorithms on CPUs (that’s what I found on their GitHub and on some articles), but I am not sure if the use RCNN actually.</p>

username: yang-gis
cooked: <p>Hello, have you successfully converted the MaskrCNN model in Torchvion to TorcScript?which version of pytorch did you use for it?</p>

username: WaterKnight
cooked: <p>Yes, you can find it in my repo:</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/WaterKnight1998/Deep-Tumour-Spheroid/blob/develop/notebooks/Mask-RCNN.ipynb" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/WaterKnight1998/Deep-Tumour-Spheroid/blob/develop/notebooks/Mask-RCNN.ipynb" target="_blank" rel="nofollow noopener">WaterKnight1998/Deep-Tumour-Spheroid/blob/develop/notebooks/Mask-RCNN.ipynb</a></h4>
<pre><code class="lang-ipynb">{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/david/anaconda3/envs/Deep-Tumour-Spheroid/lib/python3.7/site-packages/torch/nn/_reduction.py:44: UserWarning: size_average and reduce args will be deprecated, please use reduction='mean' instead.\n",
      "  warnings.warn(warning.format(ret))\n"
     ]
    }
   ],
   "source": [
    "from fastai.basics import *\n",
    "from fastai.vision import models\n",
    "from fastai.vision.all import *\n",
</code></pre>

  This file has been truncated. <a href="https://github.com/WaterKnight1998/Deep-Tumour-Spheroid/blob/develop/notebooks/Mask-RCNN.ipynb" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: morganOliveira2018
cooked: <p><a class="mention" href="/u/waterknight">@WaterKnight</a> Do you know how I generate the APK of the Torch Scripts version to run within the Android version? What type of database do I use to store model data? I’m trying to generate a version to be used on Android. Any tips? I’m using Mask-RCNN based on Detectron2.</p>

username: yorkville
cooked: <p>I just found this thread recently.  I think I am dealing with similar performance issue of deploying object detection model on CPU.</p>
<p>I am running Faster RCNN object detection model built on pytorch. The hardware is an AWS EC2 instance (T2.large with Intel Xeon, clock speed of 3.3 GHz). I am getting inference time of ~3 sec per frame. This seems to be similar to the performance the original poster (bonzogondo) mentioned above.</p>
<p>While I understand GPU is necessary for training a model, but I thought CPU is adequate for running a model in inference mode, (as I have seen real time object detection system running on raspberry pi or android phone) . Is this performance issue specific to Faster RCNN?</p>
<p>Sorry abt the newbie question. Thank you for any advice and insight.</p>

username: marksaroufim
cooked: <p>For CPU inference my recommendation is to do thing in the following order</p>
<ol>
<li>
<code>torch.set_num_threads(1)</code> <a href="https://pytorch.org/docs/stable/generated/torch.set_num_threads.html" class="inline-onebox" rel="noopener nofollow ugc">torch.set_num_threads — PyTorch 1.10 documentation</a>
</li>
<li>Picking a smaller model, distillation, quantization etc…</li>
<li>Using IPEX for accelerated CPU kernels if you’re using Intel <a href="https://github.com/intel/intel-extension-for-pytorch" rel="noopener nofollow ugc">https://github.com/intel/intel-extension-for-pytorch</a>
</li>
</ol>