Error in CUDNN status when calling batch normalization, python=3.6, pytorch=0.4.0, cuda 9

username: teresa
cooked: <p>In a project with python=3.6 and pytorch=0.4.0, I am getting the followiong error when running the model:</p>
<p>File “net.py”, line 150, in forward<br>
x = self.bn1(x)<br>
File “/home/…/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 491, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/…/lib/python3.6/site-packages/torch/nn/modules/batchnorm.py”, line 49, in forward<br>
self.training or not self.track_running_stats, self.momentum, self.eps)<br>
File “/home/…/lib/python3.6/site-packages/torch/nn/functional.py”, line 1194, in batch_norm<br>
training, momentum, eps, torch.backends.cudnn.enabled<br>
RuntimeError: CUDNN_STATUS_EXECUTION_FAILED</p>
<p>Do you know why that can be? Thank you</p>

username: teresa
cooked: <p>After some research the problem is that the RTX 3090 is not compatible with CUDA 9. Neither is the RTX 2080 compatible. Those are all my training options. Removing the CUDA references from the training code fixes the problem and right now it is training on the CPU.</p>
<p>Some research about Pytorch roadmap shows that the version 0.4.0 will never be compatible with CUDA 10 or 11 that are the GPUs in my lab.</p>
<p>Training in the CPU will take more than 50 days and it is not a realistic option.</p>
<p>Thinking about updating the Pytorch version to a one compatible with at least CUDA 10. Pytorch v. 1.0.0 would be enough.</p>
<p>Do you have any recommendation for doing this in the fastest way possible?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="teresa" data-post="2" data-topic="145004">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/teresa/48/47021_2.png" class="avatar"> teresa:</div>
<blockquote>
<p>Thinking about updating the Pytorch version to a one compatible with at least CUDA 10. Pytorch v. 1.0.0 would be enough.</p>
</blockquote>
</aside>
<p>For your 3090 you would need CUDA 11.x, so I would recommend to update to the latest stable or nightly release with CUDA 11.3 or 11.5.</p>

username: teresa
cooked: <p>Thank you. I have now updated to CUDA 11 and most of the repo works. I just have one piece that doesn’t. The full error message is is the following:</p>
<p>File “train_continue.py”, line 201, in <br>
fire.Fire()<br>
File “/home/…/lib/python3.6/site-packages/fire/core.py”, line 141, in Fire<br>
component_trace = _Fire(component, args, parsed_flag_args, context, name)<br>
File “/home/…/lib/python3.6/site-packages/fire/core.py”, line 471, in _Fire<br>
target=component.<strong>name</strong>)<br>
File “/home/…/lib/python3.6/site-packages/fire/core.py”, line 681, in _CallAndUpdateTrace<br>
component = fn(*varargs, **kwargs)<br>
File “train_continue.py”, line 120, in train<br>
loss = criterion(score_connect, target)<br>
File “/home/…/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 1102, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “/home/…/lib/python3.6/site-packages/torch/nn/modules/loss.py”, line 1152, in forward<br>
label_smoothing=self.label_smoothing)<br>
File “/home/…/lib/python3.6/site-packages/torch/nn/functional.py”, line 2846, in cross_entropy<br>
return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)<br>
RuntimeError: “nll_loss_forward_reduce_cuda_kernel_2d_index” not implemented for ‘Bool’</p>
<p>And the code around line 120 of the script is:<br>
for i, (input, target, existing) in tqdm(enumerate(train_dataloader)):</p>
<pre><code>        input = input.cuda()

        target = target.cuda()

        optimizer.zero_grad()

        score_model = model(input)

        existing = existing.cuda()

        score_model = t.cat([score_model, existing], 1)

        score_connect = connect(score_model)

        loss = criterion(score_connect, target)

        loss.backward()

        optimizer.step()
</code></pre>
<p>Do you know why this might happen?</p>

username: ptrblck
cooked: <p><code>nn.CrossEntropyLoss</code> expects logits in the shape of <code>[batch_size, nb_classes, *]</code> as a <code>FloatTensor</code> and targets in the shape <code>[batch_size, *]</code> containing class indices in the range <code>[0, nb_classes-1]</code> as a <code>LongTensor</code> (in the multi-class setup).<br>
In your use case you are using <code>bool</code> values, which is not supported.</p>