CUDNN_STATUS_MAPPING_ERROR using conv2d

pelias: <p>Would be immensely grateful for some help. New GPUs not working on my conda environment</p>
<p>In [2]: torch.version.cuda<br>
Out[2]: ‘10.1.243’</p>
<p>cudatoolkit        10.1.243       h6bb024c_0  anaconda<br>
cudnn           7.6.5.32       hc0a50b0_1  conda-forge<br>
python                    3.7.5                h0371630_0    anaconda<br>
python-dateutil           2.8.0                    py37_0    anaconda<br>
python_abi                3.7                     1_cp37m    conda-forge<br>
pytorch                   1.3.1           py3.7_cuda10.1.243_cudnn7.6.3_0    pytorch<br>
torchsummary              1.5.1                    pypi_0    pypi<br>
torchvision               0.4.2                py37_cu101    pytorch</p>
<p>i tried updating my pytorch install within my conda environment using</p>
<p><code>conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch</code></p>
<p>as well as updating cudnn<br>
and thus far no luck</p>
<p>GPUs:<br>
NVIDIA GeForce RTX 3090, 00000000:1A:00.0, 94.02.42.00.B0<br>
NVIDIA GeForce RTX 3090, 00000000:1B:00.0, 94.02.42.00.B0<br>
NVIDIA GeForce RTX 3090, 00000000:3D:00.0, 94.02.42.00.B0<br>
NVIDIA GeForce RTX 3090, 00000000:3E:00.0, 94.02.42.00.B0</p>
<p>NVIDIA-SMI 465.19.01    Driver Version: 465.19.01    CUDA Version: 11.3</p>
<pre><code class="lang-auto">
(porch) **pae2115@dendrite**:**~/ValveNet_v2**$ CUDA_IDX=0 python CADnet_ECG_001.py

View CADnet_ECG_ONLY_001_2021-11-12--07:54:27 for study log dir

[I 2021-11-12 07:54:27,246] A new study created in memory with name: CADnet_ECG_ONLY_001

data_config &lt;OptunaDataConfig(batch_size=16, shuffle=True, train_sampler=None, validation_sampler=None, data_type=2, pin_memory=True, train_features_path=/home/pae2115/CADnet/amyloid_any_typ

e_ECG_plus_echo_train_wander_removed_pct_truncated_mean_normalized_waveform_features.npy, train_labels_path=/home/pae2115/CADnet/amyloid_any_type_ECG_plus_echo_train_label.npy, train_tabular

_path=None, validation_features_path=/home/pae2115/CADnet/amyloid_any_type_ECG_plus_echo_eval_wander_removed_pct_truncated_mean_normalized_waveform_features.npy, validation_labels_path=/home

/pae2115/CADnet/amyloid_any_type_ECG_plus_echo_eval_label.npy, validation_tabular_path=None, features_permute_axes_order=[], labels_permute_axes_order=[]) @12fd0&gt;

Loading features data from /home/pae2115/CADnet/amyloid_any_type_ECG_plus_echo_train_wander_removed_pct_truncated_mean_normalized_waveform_features.npy

Loading labels data from /home/pae2115/CADnet/amyloid_any_type_ECG_plus_echo_train_label.npy

Loading tabular data from None

result shape before permute_axes_order (7364, 1, 2500, 12)

result shape before permute_axes_order (7364,)

Done loading features data from /home/pae2115/CADnet/amyloid_any_type_ECG_plus_echo_train_wander_removed_pct_truncated_mean_normalized_waveform_features.npy (7364, 1, 2500, 12)

Done loading labels data from /home/pae2115/CADnet/amyloid_any_type_ECG_plus_echo_train_label.npy (7364,)

Loading features data from /home/pae2115/CADnet/amyloid_any_type_ECG_plus_echo_eval_wander_removed_pct_truncated_mean_normalized_waveform_features.npy

Loading labels data from /home/pae2115/CADnet/amyloid_any_type_ECG_plus_echo_eval_label.npy

Loading tabular data from None

result shape before permute_axes_order (2161, 1, 2500, 12)

result shape before permute_axes_order (2161,)

Done loading features data from /home/pae2115/CADnet/amyloid_any_type_ECG_plus_echo_eval_wander_removed_pct_truncated_mean_normalized_waveform_features.npy (2161, 1, 2500, 12)
Done loading labels data from /home/pae2115/CADnet/amyloid_any_type_ECG_plus_echo_eval_label.npy (2161,)

Computing train label counts
Computing eval label counts

train label count: Counter({0.0: 5611, 1.0: 1753})
 eval label count: Counter({0.0: 1468, 1.0: 693})
POSITIVE_CLASS_WEIGHT: tensor([3.2008], dtype=torch.float64)  
High class imbalance in train, adding positive label weight(s): tensor([3.2008], dtype=torch.float64)
Optuna Train on: cuda:0
trial params dict_items([('batch_size', 16), ('lr', 5e-05), ('weight_decay', 0.01), ('optimizer_name', 'Adam'), ('filter_size', 16), ('dropout', 0.5)])
checkpoint_log_dir CADnet_ECG_ONLY_001_2021-11-12--07:54:27/trial_batch_size=16,lr=5e-05,weight_decay=0.01,optimizer_name=Adam,filter_size=16,dropout=0.5,
loss is on cuda:0 with pos weight tensor([3.2008], device='cuda:0', dtype=torch.float64)
2021-11-12 08:03:37,966 trainer INFO: Engine run starting with max_epochs=80.
2021-11-12 08:03:38,363 trainer ERROR: Current run is terminating due to exception: cuDNN error: CUDNN_STATUS_MAPPING_ERROR.
2021-11-12 08:03:38,364 trainer ERROR: Engine run is terminating due to exception: cuDNN error: CUDNN_STATUS_MAPPING_ERROR.
[W 2021-11-12 08:03:38,364] Trial 0 failed because of the following error: RuntimeError('cuDNN error: CUDNN_STATUS_MAPPING_ERROR')
Traceback (most recent call last):
  File "/home/pae2115/anaconda3/envs/porch/lib/python3.7/site-packages/optuna/study/_optimize.py", line 213, in _run_trial
    value_or_values = func(trial)
  File "/home/pae2115/ValveNet_v2/optuna_common/objective.py", line 110, in __call__
    trainer.run(train_data_loader, max_epochs=self.model_config.num_epochs)
  File "/home/pae2115/anaconda3/envs/porch/lib/python3.7/site-packages/ignite/engine/engine.py", line 691, in run
    return self._internal_run()
  File "/home/pae2115/anaconda3/envs/porch/lib/python3.7/site-packages/ignite/engine/engine.py", line 762, in _internal_run
    self._handle_exception(e)
  File "/home/pae2115/anaconda3/envs/porch/lib/python3.7/site-packages/ignite/engine/engine.py", line 467, in _handle_exception
    raise e
  File "/home/pae2115/anaconda3/envs/porch/lib/python3.7/site-packages/ignite/engine/engine.py", line 730, in _internal_run
    time_taken = self._run_once_on_dataset()
  File "/home/pae2115/anaconda3/envs/porch/lib/python3.7/site-packages/ignite/engine/engine.py", line 828, in _run_once_on_dataset
    self._handle_exception(e)
  File "/home/pae2115/anaconda3/envs/porch/lib/python3.7/site-packages/ignite/engine/engine.py", line 467, in _handle_exception
    raise e
  File "/home/pae2115/anaconda3/envs/porch/lib/python3.7/site-packages/ignite/engine/engine.py", line 811, in _run_once_on_dataset
    self.state.output = self._process_function(self, self.state.batch)
  File "/home/pae2115/anaconda3/envs/porch/lib/python3.7/site-packages/ignite/engine/__init__.py", line 99, in _update
    y_pred = model(x)
  File "/home/pae2115/anaconda3/envs/porch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/pae2115/ValveNet_v2/models/optuna_pytorch_models.py", line 84, in forward
    x = self.conv1(x)
  File "/home/pae2115/anaconda3/envs/porch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/pae2115/anaconda3/envs/porch/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 202, in forward
    self.padding, self.dilation, self.groups)
RuntimeError: cuDNN error: CUDNN_STATUS_MAPPING_ERROR

Traceback (most recent call last):
  File "CADnet_ECG_001.py", line 111, in &lt;module&gt;
    redirect=args.redirect
  File "/home/pae2115/ValveNet_v2/optuna_common/trial_utils.py", line 50, in run_trials
    study.optimize(objective, n_trials=num_trials, gc_after_trial=True)
  File "/home/pae2115/anaconda3/envs/porch/lib/python3.7/site-packages/optuna/study/study.py", line 409, in optimize
    show_progress_bar=show_progress_bar,
  File "/home/pae2115/anaconda3/envs/porch/lib/python3.7/site-packages/optuna/study/_optimize.py", line 76, in _optimize
    progress_bar=progress_bar,
  File "/home/pae2115/anaconda3/envs/porch/lib/python3.7/site-packages/optuna/study/_optimize.py", line 163, in _optimize_sequential
    trial = _run_trial(study, func, catch)
  File "/home/pae2115/anaconda3/envs/porch/lib/python3.7/site-packages/optuna/study/_optimize.py", line 264, in _run_trial
    raise func_err
  File "/home/pae2115/anaconda3/envs/porch/lib/python3.7/site-packages/optuna/study/_optimize.py", line 213, in _run_trial
    value_or_values = func(trial)
  File "/home/pae2115/ValveNet_v2/optuna_common/objective.py", line 110, in __call__
    trainer.run(train_data_loader, max_epochs=self.model_config.num_epochs)
  File "/home/pae2115/anaconda3/envs/porch/lib/python3.7/site-packages/ignite/engine/engine.py", line 691, in run
    return self._internal_run()
  File "/home/pae2115/anaconda3/envs/porch/lib/python3.7/site-packages/ignite/engine/engine.py", line 762, in _internal_run
    self._handle_exception(e)
  File "/home/pae2115/anaconda3/envs/porch/lib/python3.7/site-packages/ignite/engine/engine.py", line 467, in _handle_exception
    raise e
  File "/home/pae2115/anaconda3/envs/porch/lib/python3.7/site-packages/ignite/engine/engine.py", line 730, in _internal_run
    time_taken = self._run_once_on_dataset()
  File "/home/pae2115/anaconda3/envs/porch/lib/python3.7/site-packages/ignite/engine/engine.py", line 828, in _run_once_on_dataset
    self._handle_exception(e)
  File "/home/pae2115/anaconda3/envs/porch/lib/python3.7/site-packages/ignite/engine/engine.py", line 467, in _handle_exception
    raise e
  File "/home/pae2115/anaconda3/envs/porch/lib/python3.7/site-packages/ignite/engine/engine.py", line 811, in _run_once_on_dataset
    self.state.output = self._process_function(self, self.state.batch)
  File "/home/pae2115/anaconda3/envs/porch/lib/python3.7/site-packages/ignite/engine/__init__.py", line 99, in _update
    y_pred = model(x)
  File "/home/pae2115/anaconda3/envs/porch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/pae2115/ValveNet_v2/models/optuna_pytorch_models.py", line 84, in forward
    x = self.conv1(x)
  File "/home/pae2115/anaconda3/envs/porch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/pae2115/anaconda3/envs/porch/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 202, in forward
    self.padding, self.dilation, self.groups)
RuntimeError: cuDNN error: CUDNN_STATUS_MAPPING_ERROR
</code></pre>

ptrblck: <p>Your 3090 needs CUDA11, so you should install the current binaries with the CUDA11.3 runtime and cuDNN8.2.0.</p>

pelias: <p>This is incredibly helpful but I’m not entirely sure how to do this, could you point me to some documentation on how to do so? Thank you for always being so helpful!</p>
<p>Best,<br>
Pierre</p>

ptrblck: <p>Sure!<br>
Check the install instructions <a href="https://pytorch.org/get-started/locally/">here</a> and select “CUDA 11.3”:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/6/866636330a95c7c6a0583064c085c746218f2d8a.png" data-download-href="https://discuss.pytorch.org/uploads/default/866636330a95c7c6a0583064c085c746218f2d8a" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/6/866636330a95c7c6a0583064c085c746218f2d8a_2_690x266.png" alt="image" data-base62-sha1="jaWY3qew64ouj64oLLZhwu4FqXw" width="690" height="266" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/6/866636330a95c7c6a0583064c085c746218f2d8a_2_690x266.png, https://discuss.pytorch.org/uploads/default/original/3X/8/6/866636330a95c7c6a0583064c085c746218f2d8a.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/6/866636330a95c7c6a0583064c085c746218f2d8a.png 2x" data-dominant-color="F5DCD9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">843×325 22.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Make sure to uninstall the current PyTorch build in the used environment (run <code>pip uninstall torch -y</code> a few times and/or <code>conda uninstall pytorch -y</code>) or create a new virtual environment before installing the new binaries.</p>

pelias: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="136675">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>CUDA11</p>
</blockquote>
</aside>
<p>Thank you for the additional detail. I did this in my conda environment and then both torch and ignite become uninstalled (after running the conda install function you list above). I then run</p>
<pre><code class="lang-auto">pip install torch
</code></pre>
<p>then</p>
<pre><code class="lang-auto">conda install -c pytorch ignite
</code></pre>
<p>The latter does this</p>
<pre><code class="lang-auto">==&gt; WARNING: A newer version of conda exists. &lt;==
  current version: 4.10.1
  latest version: 4.10.3

Please update conda by running

    $ conda update -n base -c defaults conda



## Package Plan ##

  environment location: /home/pae2115/anaconda3/envs/porch

  added / updated specs:
    - ignite


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    ignite-0.4.7               |             py_0         129 KB  pytorch
    pytorch-1.7.1              |py3.7_cuda10.1.243_cudnn7.6.3_0       552.8 MB  pytorch
    ------------------------------------------------------------
                                           Total:       552.9 MB

The following NEW packages will be INSTALLED:

  ignite             pytorch/noarch::ignite-0.4.7-py_0
  libuv              pkgs/main/linux-64::libuv-1.40.0-h7b6447c_0
  **pytorch            pytorch/linux-64::pytorch-1.7.1-py3.7_cuda10.1.243_cudnn7.6.3_0**

</code></pre>
<p>It seems as though ignite overwrites to an older version of pytorch. I cant find any documentation online about pytorch ignite compatibility with CUDA 11</p>

ptrblck: <p>It seems that <code>ignite</code> is indeed pointing towards an older PyTorch release with CUDA10.<br>
Note that <code>pip install torch</code> would also install the default CUDA 10.2 runtime, so stick to the CUDA 11.3 <code>pip install</code> command.</p>
<p>In any case, once you’ve properly installed a PyTorch binaries with CUDA 11.3, try to skip the dependencies in <code>ignite</code> via <code>conda install -c pytorch --no-deps</code>.</p>
<p>CC <a class="mention" href="/u/vfdev-5">@vfdev-5</a> for visibility in case this is a known issue in <code>ignite</code></p>

pelias: <p>i attempted uninstalling pytorch and ignite  followed by</p>
<pre><code class="lang-auto">conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch-nightly
</code></pre>
<p>but after this there was no version of torch installed. Further when I check list cuda I see</p>
<pre><code class="lang-auto">cudatoolkit               10.1.243             h6bb024c_0    anaconda
</code></pre>
<p>I also tried</p>
<pre><code class="lang-auto">pip3 install torch==1.10.0+cu113 torchvision==0.11.1+cu113 torchaudio==0.10.0+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html
</code></pre>
<p>instead without luck.</p>
<p>I am now trying to build an environment from scratch.</p>
<p>And apologies for my limited understanding, but you are saying that once I get over the hurdle of pytorch installed with CUDA 11.3, then when I go to install ignite I should run</p>
<pre><code class="lang-auto">conda install -c pytorch ignite --no-deps
</code></pre>
<p>I was not sure as the conda install line you share does not have ignite in it.</p>

pelias: <p>Got it working! Had to abandon the old environment and build from scratch. This ordering (followed by all the other packages)  led to success. Thank you so much for your help!</p>
<pre><code class="lang-auto">conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch

conda list cuda

conda install -c pytorch ignite --no-deps
</code></pre>

vfdev-5: <p><a class="mention" href="/u/pelias">@pelias</a> interesting issue you are facing.<br>
I just tried to install on a new conda env based on python 3.7 and could not repro the issue:</p>
<pre><code class="lang-bash">(test) user:/# conda install pytorch torchvision cudatoolkit=11.3 -c pytorch
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /opt/conda/envs/test

  added / updated specs:
    - cudatoolkit=11.3
    - pytorch
    - torchvision


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    cudatoolkit-11.3.1         |       h2bc3f7f_2       549.3 MB
    freetype-2.11.0            |       h70c0345_0         618 KB
    giflib-5.2.1               |       h7b6447c_0          78 KB
    intel-openmp-2021.4.0      |    h06a4308_3561         4.2 MB
    jpeg-9d                    |       h7f8727e_0         232 KB
    libidn2-2.3.2              |       h7f8727e_0          81 KB
    libwebp-1.2.0              |       h89dd481_0         493 KB
    lz4-c-1.9.3                |       h295c915_1         185 KB
    mkl-2021.4.0               |     h06a4308_640       142.6 MB
    mkl-service-2.4.0          |   py37h7f8727e_0          56 KB
    mkl_fft-1.3.1              |   py37hd3c417c_0         172 KB
    mkl_random-1.2.2           |   py37h51133e4_0         287 KB
    numpy-1.21.2               |   py37h20f2e39_0          23 KB
    numpy-base-1.21.2          |   py37h79a1101_0         4.8 MB
    olefile-0.46               |           py37_0          50 KB
    pillow-8.4.0               |   py37h5aabda8_0         644 KB
    pytorch-1.10.0             |py3.7_cuda11.3_cudnn8.2.0_0        1.21 GB  pytorch
    pytorch-mutex-1.0          |             cuda           3 KB  pytorch
    torchvision-0.11.1         |       py37_cu113        30.3 MB  pytorch
    typing_extensions-3.10.0.2 |     pyh06a4308_0          31 KB
    ------------------------------------------------------------
                                           Total:        1.92 GB
</code></pre>
<p>and</p>
<pre><code class="lang-auto">(test) user:/# conda install ignite -c pytorch
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /opt/conda/envs/test

  added / updated specs:
    - ignite

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    ignite-0.4.7               |             py_0         129 KB  pytorch
    ------------------------------------------------------------
                                           Total:         129 KB

The following NEW packages will be INSTALLED:

  ignite             pytorch/noarch::ignite-0.4.7-py_0
</code></pre>
<p>Finally, I have</p>
<pre><code class="lang-auto">(test) user:/# conda list | grep pytorch
ffmpeg                    4.3                  hf484d3e_0    pytorch
ignite                    0.4.7                      py_0    pytorch
pytorch                   1.10.0          py3.7_cuda11.3_cudnn8.2.0_0    pytorch
pytorch-mutex             1.0                        cuda    pytorch
torchvision               0.11.1               py37_cu113    pytorch
</code></pre>
<p>I wonder if it is not related to the usage of conda-forge channel …</p>
<p>PS <a class="mention" href="/u/ptrblck">@ptrblck</a> thanks for pinging !</p>

ptrblck: <aside class="quote no-group" data-username="pelias" data-post="7" data-topic="136675">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/c67d28/48.png" class="avatar"> pelias:</div>
<blockquote>
<p>And apologies for my limited understanding, but you are saying that once I get over the hurdle of pytorch installed with CUDA 11.3, then when I go to install ignite I should run</p>
<pre><code class="lang-auto">conda install -c pytorch ignite --no-deps
</code></pre>
<p>I was not sure as the conda install line you share does not have ignite in it.</p>
</blockquote>
</aside>
<p>Yes, sorry for dropping the <code>ignite</code> in the install command.</p>
<p>Good to hear it’s working now.</p>
<aside class="quote no-group" data-username="vfdev-5" data-post="9" data-topic="136675">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vfdev-5/48/3535_2.png" class="avatar"> vfdev-5:</div>
<blockquote>
<p>I wonder if it is not related to the usage of conda-forge channel …</p>
</blockquote>
</aside>
<p><code>conda-forge</code> shouldn’t be needed, but you might be right that some version mismatches might be caused by a channel mix.</p>