Data become NAN after fed into GPUs

ginobilinie: <p>When I try to feed data to GPUs, I found sometimes the img data become NAN and the label data sometimes become -1, the code is below, I also paste the output of this copy of code:</p>
<pre><code>        minibatch = dataloader.next()
        imgs = minibatch['data']
        gts = minibatch['label']

        if torch.sum(torch.isnan(imgs).any()).item()&gt;0:
            print('There is NAN in imgs before sent to cuda')
        else:
            print('There is no NAN in imgs before sent to cuda')

        print('imgs:',np.amax(imgs.numpy()),np.amin(imgs.numpy()))
        print('gts:',np.unique(gts.numpy()))
        print('gts:',type(gts))
        if engine.distributed:
        # for multiple-gpu
            imgs = imgs.cuda(non_blocking=True)
            gts = gts.cuda(non_blocking=True)
            print('gts tensor on cuda:',torch.unique(gts))
            if torch.sum(torch.isnan(imgs).any()).item()&gt;0:
                print('There is NAN in imgs on cuda')

            #print('no move to cuda')
</code></pre>
<p>Output:<br>
There is no NAN in imgs before sent to cuda<br>
There is no NAN in imgs before sent to cuda<br>
There is no NAN in imgs before sent to cuda<br>
There is no NAN in imgs before sent to cuda<br>
imgs: 0.594 -0.46539214<br>
imgs: 0.594 -0.46539214<br>
imgs: 0.594 -0.44970587<br>
imgs: 0.594 -0.46147057<br>
gts: [  0   1   2   4   5   8   9  10  11  13  18 255]<br>
gts: &lt;class ‘torch.Tensor’&gt;<br>
gts: [  0   1   2   5   7   8  10  11  13  15  18 255]<br>
gts: &lt;class ‘torch.Tensor’&gt;<br>
gts tensor on cuda: tensor([  0,   1,   2,   4,   5,   8,   9,  10,  11,  13,  18, 255],<br>
device=‘cuda:3’)<br>
gts tensor: tensor([  0,   1,   2,   4,   5,   8,   9,  10,  11,  13,  18, 255],<br>
device=‘cuda:3’)<br>
gts: [  0   1   2   3   5   6   7   8   9  11  13  18 255]<br>
gts: &lt;class ‘torch.Tensor’&gt;<br>
gts tensor on cuda: tensor([  0,   1,   2,   5,   7,   8,  10,  11,  13,  15,  18, 255],<br>
device=‘cuda:1’)<br>
There is NAN in imgs on cuda<br>
gts tensor: tensor([  0,   1,   2,   5,   7,   8,  10,  11,  13,  15,  18, 255],<br>
device=‘cuda:1’)</p>
<p>Environment:<br>
python3.6<br>
torch 1.1.0<br>
torchvision 0.3.0<br>
cuda: 9.0</p>

ptrblck: <p>Could you update to the latest stable release (<code>1.5</code>) and rerun the code?<br>
Also, could you post some information about your system, e.g. which GPU(s) are you using?<br>
We had a similar issue some time ago and if I’m not mistaken, it turned out to be a hardware failure.</p>

ginobilinie: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Thank you for your reply.</p>
<p>I’m using Tesla V100 GPU on Ubuntu System.</p>
<p>I’m trying a different version of pytorch.</p>

ginobilinie: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I solve it by rebooting the machine. <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"> <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"> <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"></p>