torch.cuda.DeferredCudaCallError: A strange bug after building a cuda extension

username: Ko-vey
cooked: <p>Hi! Here I got some bugs about Pytorch and cuda.<br>
I was training my models fine with 2 NVIDIA GeForce RTX 3090 cards on a machine, with a python virtual environment created by conda. To introduce an extra model using cuda extension into my project, I compiled and built the extension running ‘’’ python setup.py install ‘’’ with python interpreter of virtual environment (Here is the compile files I used: <a href="https://github.com/MultiPath/DepthwiseConv2d" class="inline-onebox" rel="noopener nofollow ugc">GitHub - MultiPath/DepthwiseConv2d: Efficient implementation of Depthwise Conv2d. The main code is from https://github.com/MegEngine</a>). My machine uses CUDA10,2 as default, but I switched to CUDA11.0 to compile file using ‘’’ export CUDA_HOME=/usr/local/cuda-11.0 ‘’’ in SSH.<br>
Just after I compiled this extension into my virtual environment, something went wrong. When traning models, I am no longer able to use single card numbered 0 by setting ‘’’ os.environ[“CUDA_VISIBLE_DEVICES”] = ‘0’ ‘’’ in the beginning of script file on PyCharm, it will always run the single card numbered 1, the same thing happened while setting the value to ‘1’. If I want to train the model using Distributed Data Parallel (DDP) strategy with 2 cards jointly, it will report this:<br>
‘’’<br>
GPU available: True (cuda), used: True<br>
TPU available: False, using: 0 TPU cores<br>
IPU available: False, using: 0 IPUs<br>
HPU available: False, using: 0 HPUs</p>
<p>Initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/2<br>
Initializing distributed: GLOBAL_RANK: 1, MEMBER: 2/2</p>
<p>distributed_backend=nccl<br>
All distributed processes registered. Starting with 2 processes</p>
<p>xxx/xxx/envs/AST2/lib/python3.7/site-packages/pytorch_lightning/strategies/ddp.py:437: UserWarning: Error handling mechanism for deadlock detection is uninitialized. Skipping check.<br>
rank_zero_warn(“Error handling mechanism for deadlock detection is uninitialized. Skipping check.”)<br>
Traceback (most recent call last):<br>
File “xxx/xxx/envs/AST2/lib/python3.7/site-packages/torch/cuda/<strong>init</strong>.py”, line 242, in _lazy_init<br>
queued_call()<br>
File “xxx/xxx/envs/AST2/lib/python3.7/site-packages/torch/cuda/<strong>init</strong>.py”, line 125, in _check_capability<br>
capability = get_device_capability(d)<br>
File “xxx/xxx/envs/AST2/lib/python3.7/site-packages/torch/cuda/<strong>init</strong>.py”, line 357, in get_device_capability<br>
prop = get_device_properties(device)<br>
File “xxx/xxx/envs/AST2/lib/python3.7/site-packages/torch/cuda/<strong>init</strong>.py”, line 375, in get_device_properties<br>
return _get_device_properties(device) # type: ignore[name-defined]<br>
RuntimeError: device &gt;= 0 &amp;&amp; device &lt; num_gpus INTERNAL ASSERT FAILED at “…/aten/src/ATen/cuda/CUDAContext.cpp”:50, please report a bug to PyTorch.</p>
<p>The above exception was the direct cause of the following exception:</p>
<p>Traceback (most recent call last):<br>
File “xxx/xxx/myProjects/asteroid/egs/librimix/TDANetPlus2/train-birdclefmix.py”, line 148, in<br>
main(arg_dic)<br>
File “xxx/xxx/myProjects/asteroid/egs/librimix/TDANetPlus2/train-birdclefmix.py”, line 113, in main<br>
ckpt_path=None<br>
File “xxx/xxx/envs/AST2/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py”, line 609, in fit<br>
self, self._fit_impl, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path<br>
File “xxx/xxx/envs/AST2/lib/python3.7/site-packages/pytorch_lightning/trainer/call.py”, line 36, in _call_and_handle_interrupt<br>
return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)<br>
File “xxx/xxx/envs/AST2/lib/python3.7/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py”, line 88, in launch<br>
return function(*args, **kwargs)<br>
File “xxx/xxx/envs/AST2/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py”, line 650, in _fit_impl<br>
self._run(model, ckpt_path=self.ckpt_path)<br>
File “xxx/xxx/envs/AST2/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py”, line 1048, in _run<br>
self.strategy.setup_environment()<br>
File “xxx/xxx/envs/AST2/lib/python3.7/site-packages/pytorch_lightning/strategies/ddp.py”, line 153, in setup_environment<br>
super().setup_environment()<br>
File “xxx/xxx/envs/AST2/lib/python3.7/site-packages/pytorch_lightning/strategies/strategy.py”, line 131, in setup_environment<br>
self.accelerator.setup_device(self.root_device)<br>
File “xxx/xxx/envs/AST2/lib/python3.7/site-packages/pytorch_lightning/accelerators/cuda.py”, line 43, in setup_device<br>
_check_cuda_matmul_precision(device)<br>
File “xxx/xxx/envs/AST2/lib/python3.7/site-packages/lightning_fabric/accelerators/cuda.py”, line 345, in _check_cuda_matmul_precision<br>
major, _ = torch.cuda.get_device_capability(device)<br>
File “xxx/xxx/envs/AST2/lib/python3.7/site-packages/torch/cuda/<strong>init</strong>.py”, line 357, in get_device_capability<br>
prop = get_device_properties(device)<br>
File “xxx/xxx/envs/AST2/lib/python3.7/site-packages/torch/cuda/<strong>init</strong>.py”, line 371, in get_device_properties<br>
_lazy_init() # will define _get_device_properties<br>
File “xxx/xxx/envs/AST2/lib/python3.7/site-packages/torch/cuda/<strong>init</strong>.py”, line 246, in _lazy_init<br>
raise DeferredCudaCallError(msg) from e<br>
torch.cuda.DeferredCudaCallError: CUDA call failed lazily at initialization with error: device &gt;= 0 &amp;&amp; device &lt; num_gpus INTERNAL ASSERT FAILED at “…/aten/src/ATen/cuda/CUDAContext.cpp”:50, please report a bug to PyTorch.</p>
<p>CUDA call was originally invoked at:</p>
<p>[’ File “xxx/xxx/myProjects/asteroid/egs/librimix/TDANetPlus2/train-birdclefmix.py”, line 5, in \n import torch\n’, ’ File “”, line 983, in _find_and_load\n’, ’ File “”, line 967, in _find_and_load_unlocked\n’, ’ File “”, line 677, in _load_unlocked\n’, ’ File “”, line 728, in exec_module\n’, ’ File “”, line 219, in _call_with_frames_removed\n’, ’ File “xxx/xxx/envs/AST2/lib/python3.7/site-packages/torch/<strong>init</strong>.py”, line 798, in \n _C._initExtension(manager_path())\n’, ’ File “”, line 983, in _find_and_load\n’, ’ File “”, line 967, in _find_and_load_unlocked\n’, ’ File “”, line 677, in _load_unlocked\n’, ’ File “”, line 728, in exec_module\n’, ’ File “”, line 219, in _call_with_frames_removed\n’, ’ File “xxx/xxx/envs/AST2/lib/python3.7/site-packages/torch/cuda/<strong>init</strong>.py”, line 179, in \n _lazy_call(_check_capability)\n’, ’ File “xxx/xxx/envs/AST2/lib/python3.7/site-packages/torch/cuda/<strong>init</strong>.py”, line 177, in _lazy_call\n _queued_calls.append((callable, traceback.format_stack()))\n’]<br>
Traceback (most recent call last):<br>
File “xxx/xxx/envs/AST2/lib/python3.7/site-packages/torch/cuda/<strong>init</strong>.py”, line 242, in _lazy_init<br>
queued_call()<br>
File “xxx/xxx/envs/AST2/lib/python3.7/site-packages/torch/cuda/<strong>init</strong>.py”, line 125, in _check_capability<br>
capability = get_device_capability(d)<br>
File “xxx/xxx/envs/AST2/lib/python3.7/site-packages/torch/cuda/<strong>init</strong>.py”, line 357, in get_device_capability<br>
prop = get_device_properties(device)<br>
File “xxx/xxx/envs/AST2/lib/python3.7/site-packages/torch/cuda/<strong>init</strong>.py”, line 375, in get_device_properties<br>
return _get_device_properties(device) # type: ignore[name-defined]<br>
RuntimeError: device &gt;= 0 &amp;&amp; device &lt; num_gpus INTERNAL ASSERT FAILED at “…/aten/src/ATen/cuda/CUDAContext.cpp”:50, please report a bug to PyTorch.</p>
<p>The above exception was the direct cause of the following exception:</p>
<h2>
<a name="traceback-most-recent-call-last-file-xxxxxxmyprojectsasteroidegslibrimixtdanetplus2train-birdclefmixpy-line-148-in-mainarg_dic-file-xxxxxxmyprojectsasteroidegslibrimixtdanetplus2train-birdclefmixpy-line-113-in-main-ckpt_pathnone-file-xxxxxxenvsast2libpython37site-packagespytorch_lightningtrainertrainerpy-line-609-in-fit-self-self_fit_impl-model-train_dataloaders-val_dataloaders-datamodule-ckpt_path-file-xxxxxxenvsast2libpython37site-packagespytorch_lightningtrainercallpy-line-38-in-_call_and_handle_interrupt-return-trainer_fnargs-kwargs-file-xxxxxxenvsast2libpython37site-packagespytorch_lightningtrainertrainerpy-line-650-in-_fit_impl-self_runmodel-ckpt_pathselfckpt_path-file-xxxxxxenvsast2libpython37site-packagespytorch_lightningtrainertrainerpy-line-1048-in-_run-selfstrategysetup_environment-file-xxxxxxenvsast2libpython37site-packagespytorch_lightningstrategiesddppy-line-153-in-setup_environment-supersetup_environment-file-xxxxxxenvsast2libpython37site-packagespytorch_lightningstrategiesstrategypy-line-131-in-setup_environment-selfacceleratorsetup_deviceselfroot_device-file-xxxxxxenvsast2libpython37site-packagespytorch_lightningacceleratorscudapy-line-43-in-setup_device-_check_cuda_matmul_precisiondevice-file-xxxxxxenvsast2libpython37site-packageslightning_fabricacceleratorscudapy-line-345-in-_check_cuda_matmul_precision-major-_-torchcudaget_device_capabilitydevice-file-xxxxxxenvsast2libpython37site-packagestorchcudainitpy-line-357-in-get_device_capability-prop-get_device_propertiesdevice-file-xxxxxxenvsast2libpython37site-packagestorchcudainitpy-line-371-in-get_device_properties-_lazy_init-will-define-_get_device_properties-file-xxxxxxenvsast2libpython37site-packagestorchcudainitpy-line-246-in-_lazy_init-raise-deferredcudacallerrormsg-from-e-torchcudadeferredcudacallerror-cuda-call-failed-lazily-at-initialization-with-error-device-0-device-num_gpus-internal-assert-failed-at-atensrcatencudacudacontextcpp50-please-report-a-bug-to-pytorch-some-information-brought-by-osenvirontorch_show_cpp_stacktraces-1-exception-raised-from-getdeviceproperties-at-atensrcatencudacudacontextcpp50-most-recent-call-first-frame-0-c10errorerrorc10sourcelocation-stdstring-0x57-0x7f09df7d0457-in-xxxxxxenvsast2libpython37site-packagestorchliblibc10so-frame-1-c10detailtorchcheckfailchar-const-char-const-unsigned-int-char-const-0x68-0x7f09df79a4b5-in-xxxxxxenvsast2libpython37site-packagestorchliblibc10so-frame-2-atcudagetdevicepropertieslong-0x13f-0x7f0a27d41c3f-in-xxxxxxenvsast2libpython37site-packagestorchliblibtorch_cuda_cppso-frame-3-unknown-function-0xbab182-0x7f0a37e6f182-in-xxxxxxenvsast2libpython37site-packagestorchliblibtorch_pythonso-frame-4-unknown-function-0x3e5a3a-0x7f0a376a9a3a-in-xxxxxxenvsast2libpython37site-packagestorchliblibtorch_pythonso-frame-5-_pymethoddef_rawfastcallkeywords-0x237-0x4aef97-in-xxxxxxenvsast2binpython37-frame-6-xxxxxxenvsast2binpython37-0x4ae9f0-frame-7-_pyeval_evalframedefault-0x971-0x4a7651-in-xxxxxxenvsast2binpython37-frame-8-_pyfunction_fastcallkeywords-0x106-0x4b9d16-in-xxxxxxenvsast2binpython37-frame-9-xxxxxxenvsast2binpython37-0x4ae8df-frame-10-_pyeval_evalframedefault-0x971-0x4a7651-in-xxxxxxenvsast2binpython37-frame-11-_pyeval_evalcodewithname-0x201-0x4a5a81-in-xxxxxxenvsast2binpython37-frame-12-_pyfunction_fastcallkeywords-0x29c-0x4b9eac-in-xxxxxxenvsast2binpython37-frame-13-xxxxxxenvsast2binpython37-0x4ae8df-frame-14-_pyeval_evalframedefault-0x971-0x4a7651-in-xxxxxxenvsast2binpython37-frame-15-_pyfunction_fastcallkeywords-0x106-0x4b9d16-in-xxxxxxenvsast2binpython37-frame-16-xxxxxxenvsast2binpython37-0x4ae8df-frame-17-_pyeval_evalframedefault-0x971-0x4a7651-in-xxxxxxenvsast2binpython37-frame-18-_pyfunction_fastcallkeywords-0x106-0x4b9d16-in-xxxxxxenvsast2binpython37-frame-19-xxxxxxenvsast2binpython37-0x4ae8df-frame-20-_pyeval_evalframedefault-0x971-0x4a7651-in-xxxxxxenvsast2binpython37-frame-21-_pyfunction_fastcallkeywords-0x106-0x4b9d16-in-xxxxxxenvsast2binpython37-frame-22-xxxxxxenvsast2binpython37-0x4ae8df-frame-23-_pyeval_evalframedefault-0x971-0x4a7651-in-xxxxxxenvsast2binpython37-frame-24-_pyeval_evalcodewithname-0x201-0x4a5a81-in-xxxxxxenvsast2binpython37-frame-25-_pyfunction_fastcallkeywords-0x29c-0x4b9eac-in-xxxxxxenvsast2binpython37-frame-26-xxxxxxenvsast2binpython37-0x4ae8df-frame-27-_pyeval_evalframedefault-0x468a-0x4ab36a-in-xxxxxxenvsast2binpython37-frame-28-_pyfunction_fastcallkeywords-0x106-0x4b9d16-in-xxxxxxenvsast2binpython37-frame-29-xxxxxxenvsast2binpython37-0x4ae8df-frame-30-_pyeval_evalframedefault-0x971-0x4a7651-in-xxxxxxenvsast2binpython37-frame-31-_pyfunction_fastcallkeywords-0x106-0x4b9d16-in-xxxxxxenvsast2binpython37-frame-32-xxxxxxenvsast2binpython37-0x4ae8df-frame-33-_pyeval_evalframedefault-0xa9e-0x4a777e-in-xxxxxxenvsast2binpython37-frame-34-_pyfunction_fastcallkeywords-0x106-0x4b9d16-in-xxxxxxenvsast2binpython37-frame-35-xxxxxxenvsast2binpython37-0x4ae8df-frame-36-_pyeval_evalframedefault-0x468a-0x4ab36a-in-xxxxxxenvsast2binpython37-frame-37-_pyeval_evalcodewithname-0x201-0x4a5a81-in-xxxxxxenvsast2binpython37-frame-38-_pyfunction_fastcallkeywords-0x29c-0x4b9eac-in-xxxxxxenvsast2binpython37-frame-39-xxxxxxenvsast2binpython37-0x4ae8df-frame-40-_pyeval_evalframedefault-0xa9e-0x4a777e-in-xxxxxxenvsast2binpython37-frame-41-_pyeval_evalcodewithname-0x201-0x4a5a81-in-xxxxxxenvsast2binpython37-frame-42-_pyfunction_fastcallkeywords-0x29c-0x4b9eac-in-xxxxxxenvsast2binpython37-frame-43-xxxxxxenvsast2binpython37-0x4ae8df-frame-44-_pyeval_evalframedefault-0x15d6-0x4a82b6-in-xxxxxxenvsast2binpython37-frame-45-_pyeval_evalcodewithname-0x201-0x4a5a81-in-xxxxxxenvsast2binpython37-frame-46-_pyfunction_fastcalldict-0x2d7-0x4c0f57-in-xxxxxxenvsast2binpython37-frame-47-xxxxxxenvsast2binpython37-0x4c9a80-frame-48-pyobject_call-0x60-0x4c7170-in-xxxxxxenvsast2binpython37-frame-49-_pyeval_evalframedefault-0x1ea4-0x4a8b84-in-xxxxxxenvsast2binpython37-frame-50-_pyeval_evalcodewithname-0x201-0x4a5a81-in-xxxxxxenvsast2binpython37-frame-51-_pyfunction_fastcallkeywords-0x29c-0x4b9eac-in-xxxxxxenvsast2binpython37-frame-52-xxxxxxenvsast2binpython37-0x4ae8df-frame-53-_pyeval_evalframedefault-0x468a-0x4ab36a-in-xxxxxxenvsast2binpython37-frame-54-_pyeval_evalcodewithname-0x201-0x4a5a81-in-xxxxxxenvsast2binpython37-frame-55-_pyfunction_fastcallkeywords-0x29c-0x4b9eac-in-xxxxxxenvsast2binpython37-frame-56-xxxxxxenvsast2binpython37-0x4ae8df-frame-57-_pyeval_evalframedefault-0x15d6-0x4a82b6-in-xxxxxxenvsast2binpython37-frame-58-_pyfunction_fastcallkeywords-0x106-0x4b9d16-in-xxxxxxenvsast2binpython37-frame-59-xxxxxxenvsast2binpython37-0x4ae8df-frame-60-_pyeval_evalframedefault-0x971-0x4a7651-in-xxxxxxenvsast2binpython37-frame-61-_pyeval_evalcodewithname-0x201-0x4a5a81-in-xxxxxxenvsast2binpython37-frame-62-pyeval_evalcodeex-0x39-0x4a5879-in-xxxxxxenvsast2binpython37-frame-63-pyeval_evalcode-0x1b-0x54a8db-in-xxxxxxenvsast2binpython37-1" class="anchor" href="#traceback-most-recent-call-last-file-xxxxxxmyprojectsasteroidegslibrimixtdanetplus2train-birdclefmixpy-line-148-in-mainarg_dic-file-xxxxxxmyprojectsasteroidegslibrimixtdanetplus2train-birdclefmixpy-line-113-in-main-ckpt_pathnone-file-xxxxxxenvsast2libpython37site-packagespytorch_lightningtrainertrainerpy-line-609-in-fit-self-self_fit_impl-model-train_dataloaders-val_dataloaders-datamodule-ckpt_path-file-xxxxxxenvsast2libpython37site-packagespytorch_lightningtrainercallpy-line-38-in-_call_and_handle_interrupt-return-trainer_fnargs-kwargs-file-xxxxxxenvsast2libpython37site-packagespytorch_lightningtrainertrainerpy-line-650-in-_fit_impl-self_runmodel-ckpt_pathselfckpt_path-file-xxxxxxenvsast2libpython37site-packagespytorch_lightningtrainertrainerpy-line-1048-in-_run-selfstrategysetup_environment-file-xxxxxxenvsast2libpython37site-packagespytorch_lightningstrategiesddppy-line-153-in-setup_environment-supersetup_environment-file-xxxxxxenvsast2libpython37site-packagespytorch_lightningstrategiesstrategypy-line-131-in-setup_environment-selfacceleratorsetup_deviceselfroot_device-file-xxxxxxenvsast2libpython37site-packagespytorch_lightningacceleratorscudapy-line-43-in-setup_device-_check_cuda_matmul_precisiondevice-file-xxxxxxenvsast2libpython37site-packageslightning_fabricacceleratorscudapy-line-345-in-_check_cuda_matmul_precision-major-_-torchcudaget_device_capabilitydevice-file-xxxxxxenvsast2libpython37site-packagestorchcudainitpy-line-357-in-get_device_capability-prop-get_device_propertiesdevice-file-xxxxxxenvsast2libpython37site-packagestorchcudainitpy-line-371-in-get_device_properties-_lazy_init-will-define-_get_device_properties-file-xxxxxxenvsast2libpython37site-packagestorchcudainitpy-line-246-in-_lazy_init-raise-deferredcudacallerrormsg-from-e-torchcudadeferredcudacallerror-cuda-call-failed-lazily-at-initialization-with-error-device-0-device-num_gpus-internal-assert-failed-at-atensrcatencudacudacontextcpp50-please-report-a-bug-to-pytorch-some-information-brought-by-osenvirontorch_show_cpp_stacktraces-1-exception-raised-from-getdeviceproperties-at-atensrcatencudacudacontextcpp50-most-recent-call-first-frame-0-c10errorerrorc10sourcelocation-stdstring-0x57-0x7f09df7d0457-in-xxxxxxenvsast2libpython37site-packagestorchliblibc10so-frame-1-c10detailtorchcheckfailchar-const-char-const-unsigned-int-char-const-0x68-0x7f09df79a4b5-in-xxxxxxenvsast2libpython37site-packagestorchliblibc10so-frame-2-atcudagetdevicepropertieslong-0x13f-0x7f0a27d41c3f-in-xxxxxxenvsast2libpython37site-packagestorchliblibtorch_cuda_cppso-frame-3-unknown-function-0xbab182-0x7f0a37e6f182-in-xxxxxxenvsast2libpython37site-packagestorchliblibtorch_pythonso-frame-4-unknown-function-0x3e5a3a-0x7f0a376a9a3a-in-xxxxxxenvsast2libpython37site-packagestorchliblibtorch_pythonso-frame-5-_pymethoddef_rawfastcallkeywords-0x237-0x4aef97-in-xxxxxxenvsast2binpython37-frame-6-xxxxxxenvsast2binpython37-0x4ae9f0-frame-7-_pyeval_evalframedefault-0x971-0x4a7651-in-xxxxxxenvsast2binpython37-frame-8-_pyfunction_fastcallkeywords-0x106-0x4b9d16-in-xxxxxxenvsast2binpython37-frame-9-xxxxxxenvsast2binpython37-0x4ae8df-frame-10-_pyeval_evalframedefault-0x971-0x4a7651-in-xxxxxxenvsast2binpython37-frame-11-_pyeval_evalcodewithname-0x201-0x4a5a81-in-xxxxxxenvsast2binpython37-frame-12-_pyfunction_fastcallkeywords-0x29c-0x4b9eac-in-xxxxxxenvsast2binpython37-frame-13-xxxxxxenvsast2binpython37-0x4ae8df-frame-14-_pyeval_evalframedefault-0x971-0x4a7651-in-xxxxxxenvsast2binpython37-frame-15-_pyfunction_fastcallkeywords-0x106-0x4b9d16-in-xxxxxxenvsast2binpython37-frame-16-xxxxxxenvsast2binpython37-0x4ae8df-frame-17-_pyeval_evalframedefault-0x971-0x4a7651-in-xxxxxxenvsast2binpython37-frame-18-_pyfunction_fastcallkeywords-0x106-0x4b9d16-in-xxxxxxenvsast2binpython37-frame-19-xxxxxxenvsast2binpython37-0x4ae8df-frame-20-_pyeval_evalframedefault-0x971-0x4a7651-in-xxxxxxenvsast2binpython37-frame-21-_pyfunction_fastcallkeywords-0x106-0x4b9d16-in-xxxxxxenvsast2binpython37-frame-22-xxxxxxenvsast2binpython37-0x4ae8df-frame-23-_pyeval_evalframedefault-0x971-0x4a7651-in-xxxxxxenvsast2binpython37-frame-24-_pyeval_evalcodewithname-0x201-0x4a5a81-in-xxxxxxenvsast2binpython37-frame-25-_pyfunction_fastcallkeywords-0x29c-0x4b9eac-in-xxxxxxenvsast2binpython37-frame-26-xxxxxxenvsast2binpython37-0x4ae8df-frame-27-_pyeval_evalframedefault-0x468a-0x4ab36a-in-xxxxxxenvsast2binpython37-frame-28-_pyfunction_fastcallkeywords-0x106-0x4b9d16-in-xxxxxxenvsast2binpython37-frame-29-xxxxxxenvsast2binpython37-0x4ae8df-frame-30-_pyeval_evalframedefault-0x971-0x4a7651-in-xxxxxxenvsast2binpython37-frame-31-_pyfunction_fastcallkeywords-0x106-0x4b9d16-in-xxxxxxenvsast2binpython37-frame-32-xxxxxxenvsast2binpython37-0x4ae8df-frame-33-_pyeval_evalframedefault-0xa9e-0x4a777e-in-xxxxxxenvsast2binpython37-frame-34-_pyfunction_fastcallkeywords-0x106-0x4b9d16-in-xxxxxxenvsast2binpython37-frame-35-xxxxxxenvsast2binpython37-0x4ae8df-frame-36-_pyeval_evalframedefault-0x468a-0x4ab36a-in-xxxxxxenvsast2binpython37-frame-37-_pyeval_evalcodewithname-0x201-0x4a5a81-in-xxxxxxenvsast2binpython37-frame-38-_pyfunction_fastcallkeywords-0x29c-0x4b9eac-in-xxxxxxenvsast2binpython37-frame-39-xxxxxxenvsast2binpython37-0x4ae8df-frame-40-_pyeval_evalframedefault-0xa9e-0x4a777e-in-xxxxxxenvsast2binpython37-frame-41-_pyeval_evalcodewithname-0x201-0x4a5a81-in-xxxxxxenvsast2binpython37-frame-42-_pyfunction_fastcallkeywords-0x29c-0x4b9eac-in-xxxxxxenvsast2binpython37-frame-43-xxxxxxenvsast2binpython37-0x4ae8df-frame-44-_pyeval_evalframedefault-0x15d6-0x4a82b6-in-xxxxxxenvsast2binpython37-frame-45-_pyeval_evalcodewithname-0x201-0x4a5a81-in-xxxxxxenvsast2binpython37-frame-46-_pyfunction_fastcalldict-0x2d7-0x4c0f57-in-xxxxxxenvsast2binpython37-frame-47-xxxxxxenvsast2binpython37-0x4c9a80-frame-48-pyobject_call-0x60-0x4c7170-in-xxxxxxenvsast2binpython37-frame-49-_pyeval_evalframedefault-0x1ea4-0x4a8b84-in-xxxxxxenvsast2binpython37-frame-50-_pyeval_evalcodewithname-0x201-0x4a5a81-in-xxxxxxenvsast2binpython37-frame-51-_pyfunction_fastcallkeywords-0x29c-0x4b9eac-in-xxxxxxenvsast2binpython37-frame-52-xxxxxxenvsast2binpython37-0x4ae8df-frame-53-_pyeval_evalframedefault-0x468a-0x4ab36a-in-xxxxxxenvsast2binpython37-frame-54-_pyeval_evalcodewithname-0x201-0x4a5a81-in-xxxxxxenvsast2binpython37-frame-55-_pyfunction_fastcallkeywords-0x29c-0x4b9eac-in-xxxxxxenvsast2binpython37-frame-56-xxxxxxenvsast2binpython37-0x4ae8df-frame-57-_pyeval_evalframedefault-0x15d6-0x4a82b6-in-xxxxxxenvsast2binpython37-frame-58-_pyfunction_fastcallkeywords-0x106-0x4b9d16-in-xxxxxxenvsast2binpython37-frame-59-xxxxxxenvsast2binpython37-0x4ae8df-frame-60-_pyeval_evalframedefault-0x971-0x4a7651-in-xxxxxxenvsast2binpython37-frame-61-_pyeval_evalcodewithname-0x201-0x4a5a81-in-xxxxxxenvsast2binpython37-frame-62-pyeval_evalcodeex-0x39-0x4a5879-in-xxxxxxenvsast2binpython37-frame-63-pyeval_evalcode-0x1b-0x54a8db-in-xxxxxxenvsast2binpython37-1"></a>Traceback (most recent call last):<br>
File “xxx/xxx/myProjects/asteroid/egs/librimix/TDANetPlus2/train-birdclefmix.py”, line 148, in<br>
main(arg_dic)<br>
File “xxx/xxx/myProjects/asteroid/egs/librimix/TDANetPlus2/train-birdclefmix.py”, line 113, in main<br>
ckpt_path=None<br>
File “xxx/xxx/envs/AST2/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py”, line 609, in fit<br>
self, self._fit_impl, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path<br>
File “xxx/xxx/envs/AST2/lib/python3.7/site-packages/pytorch_lightning/trainer/call.py”, line 38, in _call_and_handle_interrupt<br>
return trainer_fn(<em>args, <em><em>kwargs)<br>
File “xxx/xxx/envs/AST2/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py”, line 650, in _fit_impl<br>
self._run(model, ckpt_path=self.ckpt_path)<br>
File “xxx/xxx/envs/AST2/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py”, line 1048, in _run<br>
self.strategy.setup_environment()<br>
File “xxx/xxx/envs/AST2/lib/python3.7/site-packages/pytorch_lightning/strategies/ddp.py”, line 153, in setup_environment<br>
super().setup_environment()<br>
File “xxx/xxx/envs/AST2/lib/python3.7/site-packages/pytorch_lightning/strategies/strategy.py”, line 131, in setup_environment<br>
self.accelerator.setup_device(self.root_device)<br>
File “xxx/xxx/envs/AST2/lib/python3.7/site-packages/pytorch_lightning/accelerators/cuda.py”, line 43, in setup_device<br>
_check_cuda_matmul_precision(device)<br>
File “xxx/xxx/envs/AST2/lib/python3.7/site-packages/lightning_fabric/accelerators/cuda.py”, line 345, in _check_cuda_matmul_precision<br>
major, _ = torch.cuda.get_device_capability(device)<br>
File “xxx/xxx/envs/AST2/lib/python3.7/site-packages/torch/cuda/<strong>init</strong>.py”, line 357, in get_device_capability<br>
prop = get_device_properties(device)<br>
File “xxx/xxx/envs/AST2/lib/python3.7/site-packages/torch/cuda/<strong>init</strong>.py”, line 371, in get_device_properties<br>
_lazy_init() # will define _get_device_properties<br>
File “xxx/xxx/envs/AST2/lib/python3.7/site-packages/torch/cuda/<strong>init</strong>.py”, line 246, in _lazy_init<br>
raise DeferredCudaCallError(msg) from e<br>
torch.cuda.DeferredCudaCallError: CUDA call failed lazily at initialization with error: device &gt;= 0 &amp;&amp; device &lt; num_gpus INTERNAL ASSERT FAILED at “…/aten/src/ATen/cuda/CUDAContext.cpp”:50, please report a bug to PyTorch.<br>
—(Some information brought by os.environ[“TORCH_SHOW_CPP_STACKTRACES”] = ‘1’)<br>
Exception raised from getDeviceProperties at …/aten/src/ATen/cuda/CUDAContext.cpp:50 (most recent call first):<br>
frame <span class="hashtag">#0:</span> c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f09df7d0457 in /xxx/xxx/envs/AST2/lib/python3.7/site-packages/torch/lib/libc10.so)<br>
frame <span class="hashtag">#1:</span> c10::detail::torchCheckFail(char const</em>, char const</em>, unsigned int, char const</em>) + 0x68 (0x7f09df79a4b5 in /xxx/xxx/envs/AST2/lib/python3.7/site-packages/torch/lib/libc10.so)<br>
frame <span class="hashtag">#2:</span> at::cuda::getDeviceProperties(long) + 0x13f (0x7f0a27d41c3f in /xxx/xxx/envs/AST2/lib/python3.7/site-packages/torch/lib/libtorch_cuda_cpp.so)<br>
frame <span class="hashtag">#3:</span>  + 0xbab182 (0x7f0a37e6f182 in /xxx/xxx/envs/AST2/lib/python3.7/site-packages/torch/lib/libtorch_python.so)<br>
frame <span class="hashtag">#4:</span>  + 0x3e5a3a (0x7f0a376a9a3a in /xxx/xxx/envs/AST2/lib/python3.7/site-packages/torch/lib/libtorch_python.so)<br>
frame <span class="hashtag">#5:</span> _PyMethodDef_RawFastCallKeywords + 0x237 (0x4aef97 in /xxx/xxx/envs/AST2/bin/python3.7)<br>
frame <span class="hashtag">#6:</span> /xxx/xxx/envs/AST2/bin/python3.7() [0x4ae9f0]<br>
frame <span class="hashtag">#7:</span> _PyEval_EvalFrameDefault + 0x971 (0x4a7651 in /xxx/xxx/envs/AST2/bin/python3.7)<br>
frame <span class="hashtag">#8:</span> _PyFunction_FastCallKeywords + 0x106 (0x4b9d16 in /xxx/xxx/envs/AST2/bin/python3.7)<br>
frame <span class="hashtag">#9:</span> /xxx/xxx/envs/AST2/bin/python3.7() [0x4ae8df]<br>
frame <span class="hashtag">#10:</span> _PyEval_EvalFrameDefault + 0x971 (0x4a7651 in /xxx/xxx/envs/AST2/bin/python3.7)<br>
frame <span class="hashtag">#11:</span> _PyEval_EvalCodeWithName + 0x201 (0x4a5a81 in /xxx/xxx/envs/AST2/bin/python3.7)<br>
frame <span class="hashtag">#12:</span> _PyFunction_FastCallKeywords + 0x29c (0x4b9eac in /xxx/xxx/envs/AST2/bin/python3.7)<br>
frame <span class="hashtag">#13:</span> /xxx/xxx/envs/AST2/bin/python3.7() [0x4ae8df]<br>
frame <span class="hashtag">#14:</span> _PyEval_EvalFrameDefault + 0x971 (0x4a7651 in /xxx/xxx/envs/AST2/bin/python3.7)<br>
frame <span class="hashtag">#15:</span> _PyFunction_FastCallKeywords + 0x106 (0x4b9d16 in /xxx/xxx/envs/AST2/bin/python3.7)<br>
frame <span class="hashtag">#16:</span> /xxx/xxx/envs/AST2/bin/python3.7() [0x4ae8df]<br>
frame <span class="hashtag">#17:</span> _PyEval_EvalFrameDefault + 0x971 (0x4a7651 in /xxx/xxx/envs/AST2/bin/python3.7)<br>
frame <span class="hashtag">#18:</span> _PyFunction_FastCallKeywords + 0x106 (0x4b9d16 in /xxx/xxx/envs/AST2/bin/python3.7)<br>
frame <span class="hashtag">#19:</span> /xxx/xxx/envs/AST2/bin/python3.7() [0x4ae8df]<br>
frame <span class="hashtag">#20:</span> _PyEval_EvalFrameDefault + 0x971 (0x4a7651 in /xxx/xxx/envs/AST2/bin/python3.7)<br>
frame <span class="hashtag">#21:</span> _PyFunction_FastCallKeywords + 0x106 (0x4b9d16 in /xxx/xxx/envs/AST2/bin/python3.7)<br>
frame <span class="hashtag">#22:</span> /xxx/xxx/envs/AST2/bin/python3.7() [0x4ae8df]<br>
frame <span class="hashtag">#23:</span> _PyEval_EvalFrameDefault + 0x971 (0x4a7651 in /xxx/xxx/envs/AST2/bin/python3.7)<br>
frame <span class="hashtag">#24:</span> _PyEval_EvalCodeWithName + 0x201 (0x4a5a81 in /xxx/xxx/envs/AST2/bin/python3.7)<br>
frame <span class="hashtag">#25:</span> _PyFunction_FastCallKeywords + 0x29c (0x4b9eac in /xxx/xxx/envs/AST2/bin/python3.7)<br>
frame <span class="hashtag">#26:</span> /xxx/xxx/envs/AST2/bin/python3.7() [0x4ae8df]<br>
frame <span class="hashtag">#27:</span> _PyEval_EvalFrameDefault + 0x468a (0x4ab36a in /xxx/xxx/envs/AST2/bin/python3.7)<br>
frame <span class="hashtag">#28:</span> _PyFunction_FastCallKeywords + 0x106 (0x4b9d16 in /xxx/xxx/envs/AST2/bin/python3.7)<br>
frame <span class="hashtag">#29:</span> /xxx/xxx/envs/AST2/bin/python3.7() [0x4ae8df]<br>
frame <span class="hashtag">#30:</span> _PyEval_EvalFrameDefault + 0x971 (0x4a7651 in /xxx/xxx/envs/AST2/bin/python3.7)<br>
frame <span class="hashtag">#31:</span> _PyFunction_FastCallKeywords + 0x106 (0x4b9d16 in /xxx/xxx/envs/AST2/bin/python3.7)<br>
frame <span class="hashtag">#32:</span> /xxx/xxx/envs/AST2/bin/python3.7() [0x4ae8df]<br>
frame <span class="hashtag">#33:</span> _PyEval_EvalFrameDefault + 0xa9e (0x4a777e in /xxx/xxx/envs/AST2/bin/python3.7)<br>
frame <span class="hashtag">#34:</span> _PyFunction_FastCallKeywords + 0x106 (0x4b9d16 in /xxx/xxx/envs/AST2/bin/python3.7)<br>
frame <span class="hashtag">#35:</span> /xxx/xxx/envs/AST2/bin/python3.7() [0x4ae8df]<br>
frame <span class="hashtag">#36:</span> _PyEval_EvalFrameDefault + 0x468a (0x4ab36a in /xxx/xxx/envs/AST2/bin/python3.7)<br>
frame <span class="hashtag">#37:</span> _PyEval_EvalCodeWithName + 0x201 (0x4a5a81 in /xxx/xxx/envs/AST2/bin/python3.7)<br>
frame <span class="hashtag">#38:</span> _PyFunction_FastCallKeywords + 0x29c (0x4b9eac in /xxx/xxx/envs/AST2/bin/python3.7)<br>
frame <span class="hashtag">#39:</span> /xxx/xxx/envs/AST2/bin/python3.7() [0x4ae8df]<br>
frame <span class="hashtag">#40:</span> _PyEval_EvalFrameDefault + 0xa9e (0x4a777e in /xxx/xxx/envs/AST2/bin/python3.7)<br>
frame <span class="hashtag">#41:</span> _PyEval_EvalCodeWithName + 0x201 (0x4a5a81 in /xxx/xxx/envs/AST2/bin/python3.7)<br>
frame <span class="hashtag">#42:</span> _PyFunction_FastCallKeywords + 0x29c (0x4b9eac in /xxx/xxx/envs/AST2/bin/python3.7)<br>
frame <span class="hashtag">#43:</span> /xxx/xxx/envs/AST2/bin/python3.7() [0x4ae8df]<br>
frame <span class="hashtag">#44:</span> _PyEval_EvalFrameDefault + 0x15d6 (0x4a82b6 in /xxx/xxx/envs/AST2/bin/python3.7)<br>
frame <span class="hashtag">#45:</span> _PyEval_EvalCodeWithName + 0x201 (0x4a5a81 in /xxx/xxx/envs/AST2/bin/python3.7)<br>
frame <span class="hashtag">#46:</span> _PyFunction_FastCallDict + 0x2d7 (0x4c0f57 in /xxx/xxx/envs/AST2/bin/python3.7)<br>
frame <span class="hashtag">#47:</span> /xxx/xxx/envs/AST2/bin/python3.7() [0x4c9a80]<br>
frame <span class="hashtag">#48:</span> PyObject_Call + 0x60 (0x4c7170 in /xxx/xxx/envs/AST2/bin/python3.7)<br>
frame <span class="hashtag">#49:</span> _PyEval_EvalFrameDefault + 0x1ea4 (0x4a8b84 in /xxx/xxx/envs/AST2/bin/python3.7)<br>
frame <span class="hashtag">#50:</span> _PyEval_EvalCodeWithName + 0x201 (0x4a5a81 in /xxx/xxx/envs/AST2/bin/python3.7)<br>
frame <span class="hashtag">#51:</span> _PyFunction_FastCallKeywords + 0x29c (0x4b9eac in /xxx/xxx/envs/AST2/bin/python3.7)<br>
frame <span class="hashtag">#52:</span> /xxx/xxx/envs/AST2/bin/python3.7() [0x4ae8df]<br>
frame <span class="hashtag">#53:</span> _PyEval_EvalFrameDefault + 0x468a (0x4ab36a in /xxx/xxx/envs/AST2/bin/python3.7)<br>
frame <span class="hashtag">#54:</span> _PyEval_EvalCodeWithName + 0x201 (0x4a5a81 in /xxx/xxx/envs/AST2/bin/python3.7)<br>
frame <span class="hashtag">#55:</span> _PyFunction_FastCallKeywords + 0x29c (0x4b9eac in /xxx/xxx/envs/AST2/bin/python3.7)<br>
frame <span class="hashtag">#56:</span> /xxx/xxx/envs/AST2/bin/python3.7() [0x4ae8df]<br>
frame <span class="hashtag">#57:</span> _PyEval_EvalFrameDefault + 0x15d6 (0x4a82b6 in /xxx/xxx/envs/AST2/bin/python3.7)<br>
frame <span class="hashtag">#58:</span> _PyFunction_FastCallKeywords + 0x106 (0x4b9d16 in /xxx/xxx/envs/AST2/bin/python3.7)<br>
frame <span class="hashtag">#59:</span> /xxx/xxx/envs/AST2/bin/python3.7() [0x4ae8df]<br>
frame <span class="hashtag">#60:</span> _PyEval_EvalFrameDefault + 0x971 (0x4a7651 in /xxx/xxx/envs/AST2/bin/python3.7)<br>
frame <span class="hashtag">#61:</span> _PyEval_EvalCodeWithName + 0x201 (0x4a5a81 in /xxx/xxx/envs/AST2/bin/python3.7)<br>
frame <span class="hashtag">#62:</span> PyEval_EvalCodeEx + 0x39 (0x4a5879 in /xxx/xxx/envs/AST2/bin/python3.7)<br>
frame <span class="hashtag">#63:</span> PyEval_EvalCode + 0x1b (0x54a8db in /xxx/xxx/envs/AST2/bin/python3.7)</h2>
<p>CUDA call was originally invoked at:</p>
<p>[’ File “xxx/xxx/xxx/train.py”, line 5, in \n import torch\n’, ’ File “”, line 983, in _find_and_load\n’, ’ File “”, line 967, in _find_and_load_unlocked\n’, ’ File “”, line 677, in _load_unlocked\n’, ’ File “”, line 728, in exec_module\n’, ’ File “”, line 219, in _call_with_frames_removed\n’, ’ File “xxx/xxx/envs/AST2/lib/python3.7/site-packages/torch/<strong>init</strong>.py”, line 798, in \n _C._initExtension(manager_path())\n’, ’ File “”, line 983, in _find_and_load\n’, ’ File “”, line 967, in _find_and_load_unlocked\n’, ’ File “”, line 677, in _load_unlocked\n’, ’ File “”, line 728, in exec_module\n’, ’ File “”, line 219, in _call_with_frames_removed\n’, ’ File “xxx/xxx/envs/AST2/lib/python3.7/site-packages/torch/cuda/<strong>init</strong>.py”, line 179, in \n _lazy_call(_check_capability)\n’, ’ File “xxx/xxx/envs/AST2/lib/python3.7/site-packages/torch/cuda/<strong>init</strong>.py”, line 177, in _lazy_call\n _queued_calls.append((callable, traceback.format_stack()))\n’]</p>
<p>Process finished with exit code 1</p>
<p>‘’’</p>
<p>I tried to recreate another new virtual environment and python interpreter to avoid this bug but failed after several times. The same error still exist after I reboot my machine few times, by setting ‘’’ export CUDA_HOME=/usr/local/cuda-x.x ‘’’ from 10.2 to 11.0 or ‘’’ export CUDA_VISIBLE_DEVICES=0,1 ‘’’ in ~/.bashrc file.<br>
I also tried to remove the cuda extension by using ‘’’ rm -r build ‘’’ in the directory I compiled before, but it doesn’t work even though the extention gone.<br>
Mention that by upgrading pytorch-lightning to 2.0.0 and torch to 2.x, I can use the card number 0 or 1 again, but I still can not use DDP to train and this solution doesn’t work properly in my project for package compatibility.<br>
Do you have any idea to fix this problem? Hope for reply and bug fixing, thanks!</p>
<h3>
<a name="versions-2" class="anchor" href="#versions-2"></a>Versions</h3>
<p>Collecting environment information…<br>
PyTorch version: 1.13.1+cu117<br>
Is debug build: False<br>
CUDA used to build PyTorch: 11.7<br>
ROCM used to build PyTorch: N/A</p>
<p>OS: Ubuntu 20.04.1 LTS (x86_64)<br>
GCC version: (Ubuntu 5.5.0-12ubuntu1) 5.5.0 20171010<br>
Clang version: Could not collect<br>
CMake version: Could not collect<br>
Libc version: glibc-2.17</p>
<p>Python version: 3.7.16 (default, Jan 17 2023, 22:20:44) [GCC 11.2.0] (64-bit runtime)<br>
Python platform: Linux-5.4.0-56-generic-x86_64-with-debian-bullseye-sid<br>
Is CUDA available: True<br>
CUDA runtime version: 10.2.89<br>
CUDA_MODULE_LOADING set to: LAZY<br>
GPU models and configuration:<br>
GPU 0: NVIDIA GeForce RTX 3090<br>
GPU 1: NVIDIA GeForce RTX 3090</p>
<p>Nvidia driver version: 525.105.17<br>
cuDNN version: Probably one of the following:<br>
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.2.1<br>
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.2.1<br>
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.2.1<br>
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.2.1<br>
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.2.1<br>
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.2.1<br>
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.2.1<br>
/usr/local/cuda-10.2/targets/x86_64-linux/lib/libcudnn.so.8<br>
/usr/local/cuda-10.2/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8<br>
/usr/local/cuda-10.2/targets/x86_64-linux/lib/libcudnn_adv_train.so.8<br>
/usr/local/cuda-10.2/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8<br>
/usr/local/cuda-10.2/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8<br>
/usr/local/cuda-10.2/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8<br>
/usr/local/cuda-10.2/targets/x86_64-linux/lib/libcudnn_ops_train.so.8<br>
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn.so.8<br>
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8<br>
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_adv_train.so.8<br>
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8<br>
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8<br>
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8<br>
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_ops_train.so.8<br>
HIP runtime version: N/A<br>
MIOpen runtime version: N/A<br>
Is XNNPACK available: True</p>
<p>Versions of relevant libraries:<br>
[pip3] numpy==1.21.6<br>
[pip3] numpydoc==1.5.0<br>
[pip3] pytorch-ignite==0.3.0<br>
[pip3] pytorch-lightning==1.9.5<br>
[pip3] pytorch-ranger==0.1.1<br>
[pip3] pytorch-transformers==1.2.0<br>
[pip3] torch==1.13.1<br>
[pip3] torch-optimizer==0.3.0<br>
[pip3] torch-stoi==0.1.2<br>
[pip3] torchaudio==0.13.1<br>
[pip3] torchfile==0.1.0<br>
[pip3] torchmetrics==0.11.4<br>
[conda] numpy 1.21.6 pypi_0 pypi<br>
[conda] numpydoc 1.5.0 pypi_0 pypi<br>
[conda] pytorch-lightning 1.9.5 pypi_0 pypi<br>
[conda] pytorch-ranger 0.1.1 pypi_0 pypi<br>
[conda] torch 1.13.1 pypi_0 pypi<br>
[conda] torch-optimizer 0.3.0 pypi_0 pypi<br>
[conda] torch-stoi 0.1.2 pypi_0 pypi<br>
[conda] torchaudio 0.13.1 pypi_0 pypi<br>
[conda] torchmetrics 0.11.4 pypi_0 pypi</p>

username: ptrblck
cooked: <p>I would completely uninstall all CUDA 10.2 versions since your 3090s need CUDA 11+. Once this is done you could also try to run any other CUDA application on both devices and see if this would work as I don’t think the issue is specific to PyTorch.</p>

username: Ko-vey
cooked: <p>Thanks for your advice. I made it to uninstall CUDA-10.2 completely, after that I run the script again on both devices but it doesn’t work and reported the same problem. It still can only run on card number 1. ::</p>

username: Ko-vey
cooked: <p>After uninstalling CUDA10.2 with CUDA11.0 left, I tried to install a newer CUDA11.2 via local deb. A reboot still doesn’t work in the beginning, but the next day, it threw some exceptions: some files (for example, ~/.bashrc) are read-only, and creating a virtual env in anaconda is not allowed for not enough space, which implies a damage of file system. Tried to remount and adjust but failed. However, with a reboot again, everything back to normal.</p>