THCudaCheck FAIL invalid device function

Selcuk_Yazar: <p>Hi,</p>
<p>I try to use <a href="https://github.com/fregu856/ebms_regression" class="inline-onebox" rel="noopener nofollow ugc">GitHub - fregu856/ebms_regression: Official implementation of "Energy-Based Models for Deep Probabilistic Regression" (ECCV 2020) and "How to Train Your Energy-Based Model for Regression" (BMVC 2020).</a> repo with docker file. I think everything is ok, (Ubuntu 16.04 host file docker image run with success) I know this is very old but any chance to solve it?</p>
<p>It uses masrcnn_benchmark. I installed 2020 version of apex etc. it loads data and model but doesn’t start training.</p>
<p>compute capabilities 8.6</p>
<p>Regards.</p>
<p>2023-01-31 14:55:41,056 maskrcnn_benchmark INFO: Saving config into: /root/ebms_regression/detection/checkpoints/nce+/config.yml<br>
2023-01-31 15:13:17,753 maskrcnn_benchmark INFO: Using 1 GPUs<br>
2023-01-31 15:13:17,753 maskrcnn_benchmark INFO: Namespace(config_file=‘configs/nce+_train.yaml’, distributed=False, local_rank=0, opts=[], skip_test=False)<br>
2023-01-31 15:13:17,753 maskrcnn_benchmark INFO: Collecting env info (might take some time)<br>
2023-01-31 15:13:18,383 maskrcnn_benchmark INFO:<br>
PyTorch version: 1.0.0.dev20190401<br>
Is debug build: No<br>
CUDA used to build PyTorch: 9.0.176</p>
<p>OS: Ubuntu 16.04.5 LTS<br>
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.11) 5.4.0 20160609<br>
CMake version: version 3.14.20190401-g3e12</p>
<p>Python version: 3.6<br>
Is CUDA available: Yes<br>
CUDA runtime version: 9.0.176<br>
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3060 Laptop GPU<br>
Nvidia driver version: 512.78<br>
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.4.2</p>
<p>Versions of relevant libraries:<br>
[pip3] numpy==1.19.5<br>
[pip3] torch-nightly==1.0.0.dev20190401<br>
[pip3] torchvision-nightly==0.2.1<br>
…<br>
Selected optimization level O0: Pure FP32 training.</p>
<p>Defaults for this optimization level are:<br>
enabled : True<br>
opt_level : O0<br>
cast_model_type : torch.float32<br>
patch_torch_functions : False<br>
keep_batchnorm_fp32 : None<br>
master_weights : False<br>
loss_scale : 1.0<br>
Processing user overrides (additional kwargs that are not None)…<br>
After processing overrides, optimization options are:<br>
enabled : True<br>
opt_level : O0<br>
cast_model_type : torch.float32<br>
patch_torch_functions : False<br>
keep_batchnorm_fp32 : None<br>
master_weights : False<br>
loss_scale : 1.0<br>
2023-02-05 11:30:55,867 maskrcnn_benchmark.utils.checkpoint INFO: Loading checkpoint from /root/ebms_regression/detection/pretrained_models/e2e_faster_R-50-FPN_1x.pkl<br>
2023-02-05 11:30:56,796 maskrcnn_benchmark.utils.c2_model_loading INFO: Remapping C2 weights<br>
2023-02-05 11:30:56,796 maskrcnn_benchmark.utils.c2_model_loading INFO: C2 name: bbox_pred_b mapped name: bbox_pred.bias<br>
2023-02-05 11:30:56,797 maskrcnn_benchmark.utils.c2_model_loading INFO: C2 name: bbox_pred_w mapped name: bbox_pred.weight<br>
…<br>
creating index…<br>
index created!<br>
2023-02-05 10:49:27,779 maskrcnn_benchmark.utils.miscellaneous INFO: Saving labels mapping into /root/ebms_regression/detection/checkpoints/nce+/labels.json<br>
2023-02-05 10:49:27,781 maskrcnn_benchmark.trainer INFO: Start training<br>
THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=383 error=8 : invalid device function</p>

ptrblck: <p>I don’t know which PyTorch version you are using and thus where exactly the code is failing, since <code>THCGeneral.cpp</code> was removed a while ago.<br>
Could you execute the code with the latest PyTorch release and see if you would still be running into this issue?</p>

roudancongya: <p>I had the same problem,My operating environment configuration is RTX3060 CUDA=11.1 pytorch=1.8.0,This is my error:THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=844 error=8 : invalid device function<br>
Traceback (most recent call last):<br>
File “/home/student/jzp/CBL/tools/train_net_step.py”, line 461, in main<br>
net_outputs = pcl(**input_data)<br>
File “/home/student/anaconda3/envs/CBL/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 491, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/student/jzp/CBL/lib/nn/parallel/data_parallel.py”, line 112, in forward<br>
outputs = [self.module(*inputs[0], **kwargs[0])]<br>
File “/home/student/anaconda3/envs/CBL/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 491, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/student/jzp/CBL/lib/modeling/model_builder_oicr_cbl.py”, line 209, in forward<br>
return self._forward(data, rois, labels, step, indexes, im_scales, roi, step_scale)<br>
File “/home/student/jzp/CBL/lib/modeling/model_builder_oicr_cbl.py”, line 221, in _forward<br>
blob_conv = self.Conv_Body(im_data)<br>
File “/home/student/anaconda3/envs/CBL/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 491, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/student/jzp/CBL/lib/modeling/vgg16.py”, line 112, in forward<br>
x = getattr(self, ‘conv%d’ % i)(x)<br>
File “/home/student/anaconda3/envs/CBL/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 491, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/student/anaconda3/envs/CBL/lib/python3.6/site-packages/torch/nn/modules/container.py”, line 91, in forward<br>
input = module(input)<br>
File “/home/student/anaconda3/envs/CBL/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 491, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/student/anaconda3/envs/CBL/lib/python3.6/site-packages/torch/nn/modules/conv.py”, line 301, in forward<br>
self.padding, self.dilation, self.groups)<br>
RuntimeError: CUDNN_STATUS_EXECUTION_FAILED<br>
terminate called without an active exception</p>
<p>进程已结束，退出代码为 134 (interrupted by signal 6: SIGABRT)</p>

ptrblck: <p>Update PyTorch to the latest stable or nightly release and check if you are still seeing this issue.</p>

roudancongya: <p>I tried 1.7.0 and 1.10.1，the question still comes up</p>

roudancongya: <p>But I get this error when I use 1.10.0 :Traceback (most recent call last):<br>
File “tools/train_net_step.py”, line 22, in <br>
import nn as mynn<br>
File “/home/student/jzp/CBL/lib/nn/<strong>init</strong>.py”, line 2, in <br>
from .parallel import DataParallel<br>
File “/home/student/jzp/CBL/lib/nn/parallel/<strong>init</strong>.py”, line 3, in <br>
from .data_parallel import DataParallel, data_parallel<br>
File “/home/student/jzp/CBL/lib/nn/parallel/data_parallel.py”, line 4, in <br>
from .scatter_gather import scatter_kwargs, gather<br>
File “/home/student/jzp/CBL/lib/nn/parallel/scatter_gather.py”, line 8, in <br>
from torch.utils.data.dataloader import numpy_type_map<br>
ImportError: cannot import name ‘numpy_type_map’</p>

ptrblck: <aside class="quote no-group" data-username="roudancongya" data-post="5" data-topic="171837" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/roudancongya/48/70328_2.png" class="avatar"> roudancongya:</div>
<blockquote>
<p>I tried 1.7.0 and 1.10.1，the question still comes up</p>
</blockquote>
</aside>
<p>Both of these releases are old, so update to the latest stable (2.3.0) or nightly release.</p>

roudancongya: <p>My GPU doesn’t support later versions of CUDA, so I’m stuck with pytorch=1.X</p>

ptrblck: <p>Your RTX 3060 does support all recent CUDA versions including the 12.5. The latest PyTorch stable binaries use CUDA 12.1 while the latest nightly binaries use CUDA 12.4, so you can still update.</p>

roudancongya: <p>Thank you for your reply. I will try to see if I can solve the problem. At the same time, I want to confirm whether the new version of pytorch has numpy_type_map</p>

roudancongya: <p>I forgot to add that the code I want to run is written based on pytorch=0.4.0, what will happen if I run with the latest pytorch</p>

ptrblck: <p>It’s unclear how many function calls would need to be changed and you might need to fix each warning or error separately.</p>