Whether the GeneralizedRCNN model (trained by detectron2) can be converted to Onnx with torch.onnx.export()?

Panda_BruceLee: <p>I’ve been trying for days to use torch.onnx.export() to convert my trained detectron2 model to onnx.</p>
<p>The detectron2 model is a GeneralizedRCNN model, It is also the ideal model that took me a long time to train, using my own data set.</p>
<p>Now I want to convert it to onnx to deploy it, but there are always various errors.</p>
<p>When I read the official document today, I found that it seems that the model of type “recurrent network” cannot be converted with torch.onnx.export()?  It is said that torch.jit.trace() cannot convert it.</p>
<p>Sincerely and eagerly requesting an answer, is it possible to use torch.onnx.export() to complete the conversion of the detectron2 trained  GeneralizedRCNN model?</p>
<p>I have been tortured for many days, please help me dear experts!</p>

ptrblck: <p>Based on <a href="https://github.com/facebookresearch/detectron2/issues/4414">this issue</a> it seems to be possible to export Detectron2 to ONNX. Did you already check this topic?</p>

Panda_BruceLee: <p>Still can’t achieve that …</p>
<p>1、I guarantee I followed the instructions on the README carefully. <a href="https://github.com/NVIDIA/TensorRT/tree/main/samples/python/detectron2#detectron-2-mask-r-cnn-r50-fpn-3x-in-tensorrt" class="inline-onebox" rel="noopener nofollow ugc">TensorRT/samples/python/detectron2 at main · NVIDIA/TensorRT · GitHub</a></p>
<p>2、The only difference is I used my custom data, I’m wondering if the conversion can’t be done after using the custom data?<br>
（According to my understanding, after using custom dataset, only the weight of the model has changed, but the structure of the model has not changed, so Detectron 2 → ONNX → TensorRT should work as well. I don’t know if the current conversion can only be done for Mask R-CNN R50-FPN 3x? Or we can also convert the transfer learning model with custom dataset?）.</p>
<p>3、.yaml file changed a little bit for transfer learning.</p>
<p>--------------My Code--------------------------</p>
<p><code>%run /content/detectron2/tools/deploy/export_model.py \  --sample-image /content/new_1344_1344.jpg \ --config-file  /content/output.yaml \ --export-method tracing \ --format onnx \  --output /content \ MODEL.WEIGHTS /content/output/model_final.pth \  MODEL.DEVICE cuda</code></p>
<p>--------------My Code--------------------------</p>
<p>--------------The Bug--------------------------</p>
<p>`KeyError Traceback (most recent call last)</p>
<p>/content/detectron2/tools/deploy/export_model.py in<br>
224 exported_model = export_scripting(torch_model)<br>
225 elif args.export_method == “tracing”:<br>
→ 226 exported_model = export_tracing(torch_model, sample_inputs)<br>
227<br>
228 # run evaluation with the converted model</p>
<p>8 frames</p>
<p>[/usr/local/lib/python3.8/dist-packages/torch/onnx/symbolic_opset9.py] in to(g, self, *args)<br>
1994 # aten::to(Tensor, Tensor, bool, bool, memory_format)<br>
1995 dtype = args[0].type().scalarType()<br>
→ 1996 return g.op(“Cast”, self, to_i=sym_help.cast_pytorch_to_onnx[dtype])<br>
1997 else:<br>
1998 # aten::to(Tensor, ScalarType, bool, bool, memory_format)</p>
<p>KeyError: ‘UNKNOWN_SCALAR’`</p>
<p>--------------The Bug--------------------------</p>
<p>--------------Environment Information--------------------------</p>
<p>PyTorch version: 1.10.1+cu111<br>
Is debug build: False<br>
CUDA used to build PyTorch: 11.1<br>
ROCM used to build PyTorch: N/A</p>
<p>OS: Ubuntu 20.04.5 LTS (x86_64)<br>
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0<br>
Clang version: 10.0.0-4ubuntu1<br>
CMake version: version 3.22.6<br>
Libc version: glibc-2.31</p>
<p>Python version: 3.8.10 (default, Nov 14 2022, 12:59:47) [GCC 9.4.0] (64-bit runtime)<br>
Python platform: Linux-5.10.147±x86_64-with-glibc2.29<br>
Is CUDA available: True<br>
CUDA runtime version: 11.2.152<br>
CUDA_MODULE_LOADING set to:<br>
GPU models and configuration: GPU 0: Tesla T4<br>
Nvidia driver version: 510.47.03<br>
cuDNN version: Probably one of the following:<br>
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.1.1<br>
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.1.1<br>
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.1.1<br>
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.1.1<br>
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.1.1<br>
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.1.1<br>
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.1.1<br>
HIP runtime version: N/A<br>
MIOpen runtime version: N/A<br>
Is XNNPACK available: True</p>
<p>CPU:<br>
Architecture: x86_64<br>
CPU op-mode(s): 32-bit, 64-bit<br>
Byte Order: Little Endian<br>
Address sizes: 46 bits physical, 48 bits virtual<br>
CPU(s): 2<br>
On-line CPU(s) list: 0,1<br>
Thread(s) per core: 2<br>
Core(s) per socket: 1<br>
Socket(s): 1<br>
NUMA node(s): 1<br>
Vendor ID: GenuineIntel<br>
CPU family: 6<br>
Model: 85<br>
Model name: Intel(R) Xeon(R) CPU @ 2.00GHz<br>
Stepping: 3<br>
CPU MHz: 2000.210<br>
BogoMIPS: 4000.42<br>
Hypervisor vendor: KVM<br>
Virtualization type: full<br>
L1d cache: 32 KiB<br>
L1i cache: 32 KiB<br>
L2 cache: 1 MiB<br>
L3 cache: 38.5 MiB<br>
NUMA node0 CPU(s): 0,1<br>
Vulnerability Itlb multihit: Not affected<br>
Vulnerability L1tf: Mitigation; PTE Inversion<br>
Vulnerability Mds: Vulnerable; SMT Host state unknown<br>
Vulnerability Meltdown: Vulnerable<br>
Vulnerability Mmio stale data: Vulnerable<br>
Vulnerability Retbleed: Vulnerable<br>
Vulnerability Spec store bypass: Vulnerable<br>
Vulnerability Spectre v1: Vulnerable: __user pointer sanitization and usercopy barriers only; no swapgs barriers<br>
Vulnerability Spectre v2: Vulnerable, IBPB: disabled, STIBP: disabled, PBRSB-eIBRS: Not affected<br>
Vulnerability Srbds: Not affected<br>
Vulnerability Tsx async abort: Vulnerable<br>
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves arat md_clear arch_capabilities</p>
<p>Versions of relevant libraries:<br>
[pip3] mypy-extensions==1.0.0<br>
[pip3] numpy==1.21.6<br>
[pip3] torch==1.10.1+cu111<br>
[pip3] torchaudio==0.10.1+rocm4.1<br>
[pip3] torchsummary==1.5.1<br>
[pip3] torchtext==0.14.1<br>
[pip3] torchvision==0.11.2+cu111<br>
[conda] Could not collect</p>
<p>--------------Environment Information--------------------------</p>

ptrblck: <p>Were you able to successfully run the example?<br>
If so, try to apply your changes separately and check which one might break your current workflow.</p>

Panda_BruceLee: <p>Hi ptrblck, what i am worrying now, is afraid that the conversion can’t be done after using the custom data.</p>
<p>If so, the carefully and separately trying is useless effort…</p>
<p>Do we have succeesful example that Detectron 2 → ONNX → TensorRT, with a custom data trained Detectron2 model ?</p>

ptrblck: <p>I don’t know and have thus recommended to verify the workflow first, which should work.</p>

RajUpadhyay: <p>I am sorry for a silly question but what is the cause of error if you were able to find out?<br>
Its a bit unclear to me.<br>
Is it related to opset?</p>

Panda_BruceLee: <p>Hi ptrblck, i took your advice, tried to run the example.</p>
<p>This time I didn’t use my custom data, totally followed the original workflow. ( <a href="https://github.com/NVIDIA/TensorRT/tree/main/samples/python/detectron2#detectron-2-mask-r-cnn-r50-fpn-3x-in-tensorrt" rel="noopener nofollow ugc">TensorRT/samples/python/detectron2 at main · NVIDIA/TensorRT · GitHub </a> )</p>
<p>But, I still met the same error…       KeyError: ‘UNKNOWN_SCALAR’`       (as above shows the detail)</p>
<p>I have already be stucked by this problem for a month, could you please help me…</p>

Raj_Upadhyaya: <aside class="quote no-group" data-username="Panda_BruceLee" data-post="1" data-topic="171474">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/panda_brucelee/48/56909_2.png" class="avatar"> Panda_BruceLee:</div>
<blockquote>
<p>orch.onnx.export() to convert my trained</p>
</blockquote>
</aside>
<p>Hi <a class="mention" href="/u/panda_brucelee">@Panda_BruceLee</a><br>
If you are trying to convert a detectron2 model, instead of using torch.onnx.export(),<br>
I highly recommend <strong>export_model.py</strong> which is present in tools/deploy directory of detectron2.</p>
<p>Using this, I was able to convert a detectron2 model into the onnx format just a few days back.</p>