Encountering error: Unrecognized tensor type ID: AutogradCUDA

krishna_kishore: <p>Traceback (most recent call last):<br>
File “refcoco/train_end2end.py”, line 60, in<br>
main()<br>
File “refcoco/train_end2end.py”, line 54, in main<br>
rank, model = train_net(args, config)<br>
File “/content/gdrive/My Drive/DDP/VL-BERT/refcoco/…/refcoco/function/train.py”, line 323, in train_net<br>
gradient_accumulate_steps=config.TRAIN.GRAD_ACCUMULATE_STEPS)<br>
File “/content/gdrive/My Drive/DDP/VL-BERT/refcoco/…/common/trainer.py”, line 115, in train<br>
outputs, loss = net(*batch)<br>
File “/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py”, line 727, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “/content/gdrive/My Drive/DDP/VL-BERT/refcoco/…/common/module.py”, line 22, in forward<br>
return self.train_forward(*inputs, **kwargs)<br>
File “/content/gdrive/My Drive/DDP/VL-BERT/refcoco/…/refcoco/modules/resnet_vlbert_for_refcoco.py”, line 96, in train_forward<br>
segms=None)<br>
File “/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py”, line 727, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “/content/gdrive/My Drive/DDP/VL-BERT/refcoco/…/common/fast_rcnn.py”, line 149, in forward<br>
roi_align_res = self.roi_align(img_feats[‘body4’], rois).type(images.dtype)<br>
File “/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py”, line 727, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “/content/gdrive/My Drive/DDP/VL-BERT/refcoco/…/common/lib/roi_pooling/roi_align.py”, line 69, in forward<br>
input.float(), rois.float(), self.output_size, self.spatial_scale, self.sampling_ratio<br>
File “/content/gdrive/My Drive/DDP/VL-BERT/refcoco/…/common/lib/roi_pooling/roi_align.py”, line 20, in forward<br>
input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio<br>
<strong>RuntimeError: Unrecognized tensor type ID: AutogradCUDA</strong></p>
<p>I am running on google colab, using pytorch version 1.7.0 , torchvision 0.8.1 and cuda 10.1. Same error is coming with cuda 9.2 also.</p>

ptrblck: <p>Based on the stack trace it seems you might be using a custom <code>roi_align</code> method, which is raising this issue. If so, could you check which line of code exactly is raising the error?</p>

HeyangQin: <p>I got the same error after upgrading Pytorch. It seems the problem is caused by apex compiled with the old Pytorch. After reinstalling apex from source, the problem is gone.</p>

chandrachud: <p>Hi, I have the same error. Using PyTorch 1.7 on CUDA 11.0 installed thus:</p>
<pre><code class="lang-auto">pip install torch==1.7.0+cu110 torchvision==0.8.1+cu110 -f https://download.pytorch.org/whl/torch_stable.html
</code></pre>
<p>The error comes from line 138 on this script: <a href="https://github.com/MhLiao/DB/blob/master/assets/ops/dcn/functions/deform_conv.py" rel="noopener nofollow ugc">https://github.com/MhLiao/DB/blob/master/assets/ops/dcn/functions/deform_conv.py</a><br>
which calls the CUDA function in the compiled <code>deform_conv_cuda</code>.</p>
<p>I had to make some changes to the .cpp files back when upgrading to PyTorch 1.6, but had no problems after that. This is totally new.</p>
<p>Not sure if apex is being used?</p>

kroq-gar78: <p>I also had this error after upgrading from PyTorch 1.6 to 1.7 (except with <code>AutogradCPU</code>). The codebase I’m working with has a C++ extension, and recompiling it solved the problem.</p>

Bill_Li: <p>Can you share what changes you make (do you make them in the /site-package/ path)? Thanks.</p>

chandrachud: <p>Hey, we tried using Pytorch 1.8 (nightly build), and that solved the issue.</p>

massyzs: <p>You saved my life. What you said really solved the problem confusing me about a whole week!</p>