3090 pytorch not working

username: J-bytes
cooked: <p>I keep having error on my 3090. It will crash randomly but always at some point . I’ve reproduce the error with a minimal example. I’m all out of idea tried on wsl, windows, ubuntu, I’ve tried reinstalling the drivers, pytorch, nvidia toolkit, etc … Any help would be greatly appreciated</p>
<p>Currrently using cuda toolkit 11.7 , pytorch 1.13 , nvidia driver version 525</p>
<p>import os<br>
os.environ[‘CUDA_LAUNCH_BLOCKING’] = “1”<br>
import torch<br>
torch.backends.cuda.matmul.allow_tf32 = False<br>
torch.backends.cudnn.benchmark = False<br>
torch.backends.cudnn.deterministic = False<br>
torch.backends.cudnn.allow_tf32 = True</p>
<p>for i in range(0,1000) :<br>
data = torch.randn([512, 1024, 1, 1], dtype=torch.float, device=‘cuda’, requires_grad=True)<br>
net = torch.nn.Conv2d(1024, 1024, kernel_size=[3, 3], padding=[1, 1], stride=[1, 1], dilation=[1, 1], groups=32)<br>
net = net.cuda().float()<br>
out = net(data)<br>
out.backward(torch.randn_like(out))<br>
torch.cuda.synchronize()</p>
<p>#====================================================<br>
Traceback (most recent call last):<br>
File “/home/jonathan/PycharmProjects/Adversarial_learning_paper_presentation/debug.py”, line 17, in <br>
out.backward(torch.randn_like(out))<br>
File “/home/jonathan/miniconda3/envs/Adversarial_learning_paper_presentation/lib/python3.9/site-packages/torch/_tensor.py”, line 488, in backward<br>
torch.autograd.backward(<br>
File “/home/jonathan/miniconda3/envs/Adversarial_learning_paper_presentation/lib/python3.9/site-packages/torch/autograd/<strong>init</strong>.py”, line 197, in backward<br>
Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass<br>
RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED<br>
You can try to repro this exception using the following code snippet. If that doesn’t trigger the error, please include your original repro script when reporting this issue.</p>
<p>import torch<br>
torch.backends.cuda.matmul.allow_tf32 = False<br>
torch.backends.cudnn.benchmark = False<br>
torch.backends.cudnn.deterministic = False<br>
torch.backends.cudnn.allow_tf32 = True<br>
data = torch.randn([512, 1024, 1, 1], dtype=torch.float, device=‘cuda’, requires_grad=True)<br>
net = torch.nn.Conv2d(1024, 1024, kernel_size=[3, 3], padding=[1, 1], stride=[1, 1], dilation=[1, 1], groups=32)<br>
net = net.cuda().float()<br>
out = net(data)<br>
out.backward(torch.randn_like(out))<br>
torch.cuda.synchronize()</p>
<p>ConvolutionParams<br>
memory_format = Contiguous<br>
data_type = CUDNN_DATA_FLOAT<br>
padding = [1, 1, 0]<br>
stride = [1, 1, 0]<br>
dilation = [1, 1, 0]<br>
groups = 32<br>
deterministic = false<br>
allow_tf32 = true<br>
input: TensorDescriptor 0x7f3a140ca800<br>
type = CUDNN_DATA_FLOAT<br>
nbDims = 4<br>
dimA = 512, 1024, 1, 1,<br>
strideA = 1024, 1, 1, 1,<br>
output: TensorDescriptor 0x7f3a041b0800<br>
type = CUDNN_DATA_FLOAT<br>
nbDims = 4<br>
dimA = 512, 1024, 1, 1,<br>
strideA = 1024, 1, 1, 1,<br>
weight: FilterDescriptor 0x7f3a140b8d70<br>
type = CUDNN_DATA_FLOAT<br>
tensor_format = CUDNN_TENSOR_NCHW<br>
nbDims = 4<br>
dimA = 1024, 32, 3, 3,<br>
Pointer addresses:<br>
input: 0x7f3a5ec00000<br>
output: 0x7f3a5ee00000<br>
weight: 0x7f3a5f200000<br>
Additional pointer addresses:<br>
grad_output: 0x7f3a5ee00000<br>
grad_weight: 0x7f3a5f200000<br>
Backward filter algorithm: 5</p>
<p>Process finished with exit code 1</p>

username: ptrblck
cooked: <p>I cannot reproduce the issue using <code>torch==1.13.1+cu117</code> on a 3090 on Linux.<br>
If I understand your setup correctly you are using WSL(2) with the same wheels?<br>
Was this setup working before or did you change something which caused these random crashes?</p>

username: J-bytes
cooked: <p>It never worked very well but it seems to crash faster. I’ve trie on WSL(2), ubuntu 22.10 and windows. I’m starting to think it might be a defective GPU… This exact code snippet was run on ubuntu 22.10</p>

username: ptrblck
cooked: <p>That could be the case and you could try to run some tests unrelated to PyTorch from e.g. the CUDA samples to check if these would also randomly crash.</p>

username: J-bytes
cooked: <p>Thanks for your answer . I’ve tried the cuda samples (well a lot of them anyway ) but couldn’t reproduce the issue …</p>

username: miao_liu
cooked: <p>Hi, did you solve the problem? I also met it using cuda toolkit 11.7, pytorch 1.13, nvidia driver version 525 on ubuntu 20.04.</p>
<p>python -m torch.utils.collect_env</p>
<p>Collecting environment information…<br>
PyTorch version: 1.13.1<br>
Is debug build: False<br>
CUDA used to build PyTorch: 11.7<br>
ROCM used to build PyTorch: N/A</p>
<p>OS: Ubuntu 20.04.4 LTS (x86_64)<br>
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0<br>
Clang version: Could not collect<br>
CMake version: Could not collect<br>
Libc version: glibc-2.31</p>
<p>Python version: 3.9.16 | packaged by conda-forge | (main, Feb  1 2023, 21:39:03)  [GCC 11.3.0] (64-bit runtime)<br>
Python platform: Linux-5.15.0-67-generic-x86_64-with-glibc2.31<br>
Is CUDA available: True<br>
CUDA runtime version: Could not collect<br>
CUDA_MODULE_LOADING set to: LAZY<br>
GPU models and configuration:<br>
GPU 0: NVIDIA GeForce RTX 3090<br>
GPU 1: NVIDIA GeForce RTX 3090<br>
GPU 2: NVIDIA GeForce RTX 3090<br>
GPU 3: NVIDIA GeForce RTX 3090</p>
<p>Nvidia driver version: 525.89.02<br>
cuDNN version: Could not collect<br>
HIP runtime version: N/A<br>
MIOpen runtime version: N/A<br>
Is XNNPACK available: True</p>
<p>Versions of relevant libraries:<br>
[pip3] numpy==1.23.5<br>
[pip3] pytorch-lightning==1.7.7<br>
[pip3] torch==1.13.1<br>
[pip3] torchaudio==0.13.1<br>
[pip3] torchmetrics==0.11.4<br>
[pip3] torchvision==0.14.1<br>
[conda] blas                      1.0                         mkl    defaults<br>
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch<br>
[conda] mkl                       2021.4.0           h06a4308_640    defaults<br>
[conda] mkl-service               2.4.0            py39h7e14d7c_0    conda-forge<br>
[conda] mkl_fft                   1.3.1            py39h0c7bc48_1    conda-forge<br>
[conda] mkl_random                1.2.2            py39hde0f152_0    conda-forge<br>
[conda] numpy                     1.23.5           py39h14f4228_0    defaults<br>
[conda] numpy-base                1.23.5           py39h31eccc5_0    defaults<br>
[conda] pytorch                   1.13.1          py3.9_cuda11.7_cudnn8.5.0_0    pytorch<br>
[conda] pytorch-cuda              11.7                 h778d358_3    pytorch<br>
[conda] pytorch-lightning         1.7.7                    pypi_0    pypi<br>
[conda] pytorch-mutex             1.0                        cuda    pytorch<br>
[conda] torchaudio                0.13.1               py39_cu117    pytorch<br>
[conda] torchmetrics              0.11.4                   pypi_0    pypi<br>
[conda] torchvision               0.14.1               py39_cu117    pytorch</p>

username: J-bytes
cooked: <p>I finally just sent my GPU to the manufacturer and they have confirmed it to have failed some of their tests. The GPU was simply defective</p>