Inference on CPU for detectron2

Sylvain_Ard: <p>Hello all !<br>
I have trained my model on GPU with maskrcnn and ResNet 101 on <a href="https://github.com/facebookresearch/detectron2" class="inline-onebox" rel="noopener nofollow ugc">GitHub - facebookresearch/detectron2: Detectron2 is FAIR's next-generation platform for object detection, segmentation and other visual recognition tasks.</a> code and want now to deploy it on CPU for inference. The code is for GPU, how to translate it to CPU please ? I use demo/demo.py for test/inference.<br>
Thank you very much<br>
Best regards</p>

ptrblck: <p>I would assume there is no hard-coded dependency on CUDA in the repository so unless you manually push the data and model to the GPU, the CPU should be used.<br>
In case you are using an already provided inference script and cannot see how the GPU is used, mask it via <code>CUDA_VISIBLE_DEVICES="" python inference.py</code> so that PyTorch won’t be able to use it.</p>

Sylvain_Ard: <p>OK thank you but for installing the packages for the code, we need to install cudatoolkit ?</p>

Sylvain_Ard: <p>and I think there are hard-coded dependencies of CUDA</p>

ptrblck: <p>Could you post a minimal, executable code showing this dependency, i.e. the failure on a CPU-only node (or by hiding the GPUs via the aforementioned env variable)?</p>

Sylvain_Ard: <p>I tried to install the environment in my device without GPU, but it install cudatoolkit even if I don’t tell it to install and fails with error :<br>
tk-8.6.11            | 3.8 MB    | ################################################################### | 100%<br>
lz4-c-1.9.3          | 135 KB    | ################################################################### | 100%<br>
Preparing transaction: done<br>
Verifying transaction: done<br>
Executing transaction: done<br>
ERROR conda.core.link:_execute(698): An error occurred while installing package ‘conda-forge::cudatoolkit-11.1.1-heb2d755_7’.<br>
Rolling back transaction: done</p>
<p>LinkError: post-link script failed for package conda-forge::cudatoolkit-11.1.1-heb2d755_7<br>
location of failed script: C:\Users\Sylvain ARD.conda\envs\detectron2\Scripts.cudatoolkit-post-link.bat<br>
==&gt; script messages &lt;==<br>
<br>
==&gt; script output &lt;==<br>
stdout:<br>
stderr: Accès refusé.</p>
<p>return code: 1</p>
<p>()</p>

ptrblck: <p>I can’t see what exactly is failing, but the install instructions provide CPU-only binaries <a href="https://detectron2.readthedocs.io/en/latest/tutorials/install.html#install-pre-built-detectron2-linux-only">here</a>, so I don’t think that CUDA is needed.</p>

Sylvain_Ard: <p>I fixed this error by doing :</p><aside class="quote quote-modified" data-post="13" data-topic="110511">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sylvain_ard/48/30714_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/error-access-denied/110511/13">Error access denied</a> 
  </div>
  <blockquote>
    I had finally succeeded by doing : 
edit : 
 C:\Users\Sylvain ARD\.conda\pkgs\cudatoolkit-10.0.130-h5327add_6\Scripts\.cudatoolkit-post-link.bat 

: 
and replace : 
rem post install EULA message; `pre-link.bat` does not show message and shows warning
echo "By downloading and using the CUDA Toolkit conda packages, you accept the terms and conditions of the CUDA End User License Agreement (EULA): https://docs.nvidia.com/cuda/eula/index.html" &gt;&gt; %PREFIX%\.messages.txt

by : 
rem post install EULA m…
  </blockquote>
</aside>
<p>
I progress…</p>

Sylvain_Ard: <p>I have this error in demo/demo.py :<br>
(detectron2) C:\detectron2-main&gt;python demo/demo.py --config-file configs/COCO-InstanceSegmentation/mask_rcnn_X_101_32x8d_FPN_3x.yaml --input “datasets/trainval/image (13).jpg” --output output.jpg --confidence-threshold 0.8<br>
[09/21 11:28:20 detectron2]: Arguments: Namespace(confidence_threshold=0.8, config_file=‘configs/COCO-InstanceSegmentation/mask_rcnn_X_101_32x8d_FPN_3x.yaml’, input=[‘datasets/trainval/image (13).jpg’], opts=[], output=‘output.jpg’, video_input=None, webcam=False)<br>
Traceback (most recent call last):<br>
File “demo/demo.py”, line 117, in <br>
demo = VisualizationDemo(cfg)<br>
File “C:\detectron2-main\demo\predictor.py”, line 35, in <strong>init</strong><br>
self.predictor = DefaultPredictor(cfg)<br>
File “c:\detectron2-main\detectron2\engine\defaults.py”, line 281, in <strong>init</strong><br>
self.model = build_model(self.cfg)<br>
File “c:\detectron2-main\detectron2\modeling\meta_arch\build.py”, line 23, in build_model<br>
model.to(torch.device(cfg.MODEL.DEVICE))<br>
File “C:\Users\Sylvain ARD.conda\envs\detectron2\lib\site-packages\torch\nn\modules\module.py”, line 673, in to<br>
return self._apply(convert)<br>
File “C:\Users\Sylvain ARD.conda\envs\detectron2\lib\site-packages\torch\nn\modules\module.py”, line 387, in _apply<br>
module._apply(fn)<br>
File “C:\Users\Sylvain ARD.conda\envs\detectron2\lib\site-packages\torch\nn\modules\module.py”, line 387, in _apply<br>
module._apply(fn)<br>
File “C:\Users\Sylvain ARD.conda\envs\detectron2\lib\site-packages\torch\nn\modules\module.py”, line 409, in <em>apply<br>
param_applied = fn(param)<br>
File “C:\Users\Sylvain ARD.conda\envs\detectron2\lib\site-packages\torch\nn\modules\module.py”, line 671, in convert<br>
return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)<br>
File "C:\Users\Sylvain ARD.conda\envs\detectron2\lib\site-packages\torch\cuda_<em>init</em></em>.py", line 170, in _lazy_init<br>
torch._C._cuda_init()<br>
RuntimeError: CUDA driver initialization failed, you might not have a CUDA gpu.</p>
<p>(detectron2) C:\detectron2-main&gt;pause<br>
Appuyez sur une touche pour continuer…</p>

Sylvain_Ard: <p>apparently I need to change cfg.MODEL.DEVICE but what I have to put in ?</p>

Sylvain_Ard: <p>I finally succeeded by adding cfg.MODEL.DEVICE = ‘cpu’</p>

Sylvain_Ard: <p>this is resolved<br>
thank you</p>

Aakash_Gupta: <p><a class="mention" href="/u/sylvain_ard">@Sylvain_Ard</a> - whats the impact on your inference time when you run it on a CPU? For my use case the inference time was shooting up (<em>&gt;2-3mins per image vs msec when done on a GPU</em>) Which didn’t make sense for any practical (<em>real-life</em>) deployment</p>
<p>Did you check the D2go library? -  <a href="https://github.com/facebookresearch/d2go" class="inline-onebox" rel="noopener nofollow ugc">GitHub - facebookresearch/d2go: D2Go is a toolkit for efficient deep learning</a></p>

Sylvain_Ard: <p>for me about 40 seconds in CPU and 4 seconds in GPU, but it depends on your CPU model</p>