Pytorhc1.6+nvidia-driver418+2080ti results in:RuntimeError: CUDA error: an illegal memory access was encountered

N_Hunter: <p>Hello every,  I always encountered this <code>illegal memory access</code> bug no matter what project I trained</p>
<h1>bug description</h1>
<p>I’ll use one project as example to describe the bug:</p>
<pre><code class="lang-auto"> 1.   git clone git@github.com:weiaicunzai/pytorch-cifar100.git
 2.   cd pytorch-cifar100
 3.   python train.py -net vgg16 -gpu
</code></pre>
<p>Then I get:</p>
<pre><code class="lang-auto">Files already downloaded and verified
Files already downloaded and verified
/home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/jit/__init__.py:1119: TracerWarning: Output nr 1. of the traced function does not match the corresponding output of the Python function. Detailed error:
With rtol=1e-05 and atol=1e-05, found 3 element(s) (out of 100) whose difference(s) exceeded the margin of error (including 0 nan comparisons). The greatest difference was 1717986918400.0 (8.985408308668006e+16 vs. 8.985236509976166e+16), which occurred at index (0, 67).
  check_tolerance, strict, _force_outplace, True, _module_class)
/home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/optim/lr_scheduler.py:123: UserWarning: Detected call of `lr_scheduler.step()` before `optimizer.step()`. In PyTorch 1.1.0 and later, you should call them in the opposite order: `optimizer.step()` before `lr_scheduler.step()`.  Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
  "https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate", UserWarning)
Training Epoch: 1 [128/50000]   Loss: 4.7142    LR: 0.000256
Training Epoch: 1 [256/50000]   Loss: 4.7673    LR: 0.000512
Training Epoch: 1 [384/50000]   Loss: 4.6759    LR: 0.000767
Training Epoch: 1 [512/50000]   Loss: 4.6476    LR: 0.001023
Training Epoch: 1 [640/50000]   Loss: 4.7125    LR: 0.001279
Training Epoch: 1 [768/50000]   Loss: 4.7280    LR: 0.001535
Training Epoch: 1 [896/50000]   Loss: 4.7359    LR: 0.001790
Training Epoch: 1 [1024/50000]  Loss: 4.7307    LR: 0.002046
Training Epoch: 1 [1152/50000]  Loss: 4.6513    LR: 0.002302
Training Epoch: 1 [1280/50000]  Loss: 4.6054    LR: 0.002558
Training Epoch: 1 [1408/50000]  Loss: 4.6228    LR: 0.002813
Training Epoch: 1 [1536/50000]  Loss: 4.6426    LR: 0.003069
Traceback (most recent call last):
  File "train.py", line 209, in &lt;module&gt;
    train(epoch)
  File "train.py", line 52, in train
    writer.add_scalar('LastLayerGradients/grad_norm2_weights', para.grad.norm(), n_iter)
  File "/home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/utils/tensorboard/writer.py", line 344, in add_scalar
    scalar(tag, scalar_value), global_step, walltime)
  File "/home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/utils/tensorboard/summary.py", line 181, in scalar
    scalar = make_np(scalar)
  File "/home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/utils/tensorboard/_convert_np.py", line 28, in make_np
    return _prepare_pytorch(x)
  File "/home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/utils/tensorboard/_convert_np.py", line 36, in _prepare_pytorch
    x = x.cpu().numpy()
RuntimeError: CUDA error: an illegal memory access was encountered
</code></pre>
<p>But the same code works fine on Google colab.Then I tried to run <code>CUDA_LAUNCH_BLOCKING=1  python train.py -net resnet18 -gpu</code>:</p>
<p>gives me this:</p>
<pre><code class="lang-auto">Files already downloaded and verified
Files already downloaded and verified
/home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/jit/__init__.py:1119: TracerWarning: Output nr 1. of the traced function does not match the corresponding output of the Python function. Detailed error:
With rtol=1e-05 and atol=1e-05, found 1 element(s) (out of 100) whose difference(s) exceeded the margin of error (including 0 nan comparisons). The greatest difference was 2779565395017728.0 (-2.68501056162248e+20 vs. -2.6850383572764302e+20), which occurred at index (0, 48).
  check_tolerance, strict, _force_outplace, True, _module_class)
/home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/optim/lr_scheduler.py:123: UserWarning: Detected call of `lr_scheduler.step()` before `optimizer.step()`. In PyTorch 1.1.0 and later, you should call them in the opposite order: `optimizer.step()` before `lr_scheduler.step()`.  Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
  "https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate", UserWarning)
Training Epoch: 1 [128/50000]   Loss: 4.8053    LR: 0.000256
Training Epoch: 1 [256/50000]   Loss: 4.7550    LR: 0.000512
Training Epoch: 1 [384/50000]   Loss: 4.7402    LR: 0.000767
Training Epoch: 1 [512/50000]   Loss: 4.7109    LR: 0.001023
Training Epoch: 1 [640/50000]   Loss: 4.6995    LR: 0.001279
Training Epoch: 1 [768/50000]   Loss: 4.7558    LR: 0.001535
Training Epoch: 1 [896/50000]   Loss: 4.6715    LR: 0.001790
Training Epoch: 1 [1024/50000]  Loss: 4.7487    LR: 0.002046
Training Epoch: 1 [1152/50000]  Loss: 4.6628    LR: 0.002302
Training Epoch: 1 [1280/50000]  Loss: 4.6444    LR: 0.002558
Training Epoch: 1 [1408/50000]  Loss: 4.6350    LR: 0.002813
Training Epoch: 1 [1536/50000]  Loss: 4.5711    LR: 0.003069
Training Epoch: 1 [1664/50000]  Loss: 4.6021    LR: 0.003325
Traceback (most recent call last):
  File "train.py", line 209, in &lt;module&gt;
    train(epoch)
  File "train.py", line 44, in train
    loss.backward()
  File "/home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/tensor.py", line 185, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/autograd/__init__.py", line 127, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED
Exception raised from operator() at /opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cudnn/Conv.cpp:980 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x4d (0x7f790a3ec77d in /home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #1: &lt;unknown function&gt; + 0xcad302 (0x7f790b4ff302 in /home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
frame #2: &lt;unknown function&gt; + 0xcaf235 (0x7f790b501235 in /home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
frame #3: &lt;unknown function&gt; + 0xcaa48e (0x7f790b4fc48e in /home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
frame #4: &lt;unknown function&gt; + 0xcac07b (0x7f790b4fe07b in /home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
frame #5: at::native::cudnn_convolution_backward_input(c10::ArrayRef&lt;long&gt;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long, bool, bool) + 0xb2 (0x7f790b4fe5d2 in /home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
frame #6: &lt;unknown function&gt; + 0xd117db (0x7f790b5637db in /home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
frame #7: &lt;unknown function&gt; + 0xd415f8 (0x7f790b5935f8 in /home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
frame #8: at::cudnn_convolution_backward_input(c10::ArrayRef&lt;long&gt;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long, bool, bool) + 0x1ad (0x7f793d7b8ced in /home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #9: at::native::cudnn_convolution_backward(at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long, bool, bool, std::array&lt;bool, 2ul&gt;) + 0x223 (0x7f790b4fcca3 in /home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
frame #10: &lt;unknown function&gt; + 0xd118c5 (0x7f790b5638c5 in /home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
frame #11: &lt;unknown function&gt; + 0xd41654 (0x7f790b593654 in /home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
frame #12: at::cudnn_convolution_backward(at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long, bool, bool, std::array&lt;bool, 2ul&gt;) + 0x1e2 (0x7f793d7c76a2 in /home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #13: &lt;unknown function&gt; + 0x2c250c2 (0x7f793f48b0c2 in /home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #14: &lt;unknown function&gt; + 0x2c39684 (0x7f793f49f684 in /home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #15: at::cudnn_convolution_backward(at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long, bool, bool, std::array&lt;bool, 2ul&gt;) + 0x1e2 (0x7f793d7c76a2 in /home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #16: torch::autograd::generated::CudnnConvolutionBackward::apply(std::vector&lt;at::Tensor, std::allocator&lt;at::Tensor&gt; &gt;&amp;&amp;) + 0x258 (0x7f793f312098 in /home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #17: &lt;unknown function&gt; + 0x30d1017 (0x7f793f937017 in /home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #18: torch::autograd::Engine::evaluate_function(std::shared_ptr&lt;torch::autograd::GraphTask&gt;&amp;, torch::autograd::Node*, torch::autograd::InputBuffer&amp;, std::shared_ptr&lt;torch::autograd::ReadyQueue&gt; const&amp;) + 0x1400 (0x7f793f932860 in /home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #19: torch::autograd::Engine::thread_main(std::shared_ptr&lt;torch::autograd::GraphTask&gt; const&amp;) + 0x451 (0x7f793f933401 in /home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #20: torch::autograd::Engine::thread_init(int, std::shared_ptr&lt;torch::autograd::ReadyQueue&gt; const&amp;, bool) + 0x89 (0x7f793f92b579 in /home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #21: torch::autograd::python::PythonEngine::thread_init(int, std::shared_ptr&lt;torch::autograd::ReadyQueue&gt; const&amp;, bool) + 0x4a (0x7f7943c5a13a in /home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #22: &lt;unknown function&gt; + 0xc819d (0x7f794679019d in /home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/lib/../../../.././libstdc++.so.6)
frame #23: &lt;unknown function&gt; + 0x76db (0x7f796988a6db in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #24: clone + 0x3f (0x7f79695b3a3f in /lib/x86_64-linux-gnu/libc.so.6)
</code></pre>
<h1>Environment</h1>
<p>Here is my current env:</p>
<pre><code class="lang-auto">Collecting environment information...
PyTorch version: 1.6.0
Is debug build: False
CUDA used to build PyTorch: 10.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: Could not collect

Python version: 3.6 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: 
GPU 0: GeForce RTX 2080 Ti
GPU 1: GeForce RTX 2080 Ti
GPU 2: GeForce RTX 2080 Ti
GPU 3: GeForce RTX 2080 Ti

Nvidia driver version: 418.152.00
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.2
[pip3] torch==1.6.0
[pip3] torchvision==0.7.0
[conda] blas                      1.0                         mkl    defaults
[conda] cudatoolkit               10.1.243             h6bb024c_0    defaults
[conda] mkl                       2020.2                      256    defaults
[conda] mkl-service               2.3.0            py36he904b0f_0    defaults
[conda] mkl_fft                   1.2.0            py36h23d657b_0    defaults
[conda] mkl_random                1.1.1            py36h0573a6f_0    defaults
[conda] numpy                     1.19.2           py36h54aff64_0    defaults
[conda] numpy-base                1.19.2           py36hfa32c7d_0    defaults
[conda] pytorch                   1.6.0           py3.6_cuda10.1.243_cudnn7.6.3_0    pytorch
[conda] torchvision               0.7.0                py36_cu101    pytorch
</code></pre>
<p>I’ve also posted an issue in <a href="https://github.com/pytorch/pytorch/issues/46974" rel="noopener nofollow ugc">Github repository</a>, but didn’t seem to get an solution, so could some one tells me where I could possiblely  go wrong? Thanks.</p>