Sending data to device in Pytorch causes it to contain nan values (rocm)

dunedin87: <p>Hello,</p>
<p>A bit of background,<br>
Pytorch Version 2.0 ROCm version 5.4.2 on Linux Mint 20.3 (Ubuntu 20.04). Single AMD 6950 XT.</p>
<p>Initiating checking for cuda device results in the following warning.<br>
<code>~...local/lib/python3.8/site-packages/torch/cuda/__init__.py:546: UserWarning: Can't initialize NVML warnings.warn("Can't initialize NVML")</code></p>
<p><code>torch.cuda.is_available()</code> comes out as <code>True</code></p>
<p><code>print(torch.cuda.get_device_name(device=device))</code> shows AMD Radeon Graphics</p>
<p>Onto to the problem, my loss and model outputs were showing nan, puzzled I went back and checked the data for nan, which also showed it to be True. This was odd as I had previously run the code on Kaggle notebooks, which didn’t show it to be a problem. I then decided to test it with random data and the data was still showing it containing nan. I then checked it before sending it <code>.to(device)</code> and after, and as expected initial data did not contain nan values but after <code>.to(device)</code> it did. Following is the code for testing.</p>
<pre><code class="lang-auto">device = torch.device("cuda" if (torch.cuda.is_available()) else "cpu")
model = ENet(3,2)
model.to(device)
criterion = nn.BCELoss()
opt = torch.optim.Adam(model.parameters(), lr = 1e-4)

for i in range(10):

  data = torch.rand(3,3,512,512).float()
  print('Initial Data nan', torch.any(torch.isnan(data)))
  labels =torch.randint(low = 0, high = 2, size = (3,2)).float()
  data = data.to(device)
  print('Device Data nan',torch.any(torch.isnan(data)))
  labels = labels.to(device)

  output = model(data)
  loss = criterion(output, labels)
  loss.backward()
  opt.step()
  opt.zero_grad()
</code></pre>
<p>The output for the first loop is</p>
<pre><code class="lang-auto">Initial Data nan tensor(False)

Device Data nan tensor(True, device='cuda:0')
</code></pre>
<p>Telling me that the randomly generated data doesn’t contain nan, but after .to(device) it does.</p>
<p>The problem is agnostic to the model or data (even doing to randomly generated data), but the following is a basic model for anyone wanting to replicate and troubleshoot. Uses the package timm for efficientnet. I have tried it on a Resnet model (not from timm but locally coded) with the same error.</p>
<pre><code class="lang-auto">class ENet(nn.Module):
    def __init__(self, in_channels, num_classes):
        super(ENet, self).__init__()
        
        self.first_conv = nn.Conv2d(in_channels = in_channels, out_channels = 3, kernel_size = 1)
        self.backbone = timm.create_model('efficientnet_b0', pretrained = False)
        self.relu = nn.ReLU()
        self.classifier = nn.Linear(1000,num_classes)
        self.softmax = nn.Softmax(dim = -1)
        
    def forward(self, x):
        out = self.first_conv(x)
        out = self.relu(out)
        out = self.backbone(out)
        out = self.relu(out)
        out = self.classifier(out)
        out = self.softmax(out)
        return out
</code></pre>
<p>I have seen some other threads regarding data turning to nan when sending to device, but they were all on Cuda devices (nvidia) and none of the solutions reached there seemed to be working here. Also, as mentioned before, when I try the above code on a Kaggle notebook, which uses P100, it works, making it likely that the problem is with ROCm?<br>
What is weird is that the problem isn’t even model weights / outputs or losses, but simple data transferring to device making it have nans.</p>
<p>Other topics wtih similar problems for reference. Any help would be appreciated, thanks.</p>
<aside class="quote quote-modified" data-post="1" data-topic="78163">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ginobilinie/48/3548_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/data-become-nan-after-fed-into-gpus/78163">Data become NAN after fed into GPUs</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    When I try to feed data to GPUs, I found sometimes the img data become NAN and the label data sometimes become -1, the code is below, I also paste the output of this copy of code: 
        minibatch = dataloader.next()
        imgs = minibatch['data']
        gts = minibatch['label']

        if torch.sum(torch.isnan(imgs).any()).item()&gt;0:
            print('There is NAN in imgs before sent to cuda')
        else:
            print('There is no NAN in imgs before sent to cuda')

        print('i…
  </blockquote>
</aside>
<aside class="quote quote-modified" data-post="1" data-topic="128731">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/63days/48/41015_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/the-input-data-sometimes-becomes-nan-after-cuda/128731">The input data sometimes becomes NaN after .cuda()</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    Hi, all. I’m struggling a NaN issue. Any thoughts would be appreciated. 
When I feed the point cloud data to GPU by calling .cuda(), it becomes NaN very rarely. Code used for debugging is below. 
import torch
import h5py
from torch.utils.data import DataLoader, Dataset

class PCDataset(Dataset):
    def __init__(self):
        self.pc_data = h5py.File("shapenet.h5", "r")["data"][:, :2048]

    def __getitem__(self, index):
        return self.pc_data[index]

    def __len__(self):
        return…
  </blockquote>
</aside>