Dataset input Nan but fine on index

username: Jason1995
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/e/ce3b73d32bebb5e0b9d6f18c08782be18ad9510d.png" data-download-href="https://discuss.pytorch.org/uploads/default/ce3b73d32bebb5e0b9d6f18c08782be18ad9510d" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/e/ce3b73d32bebb5e0b9d6f18c08782be18ad9510d.png" alt="image" data-base62-sha1="tqpJA1WgpvO5WVRy3xYEDAwjMYt" width="690" height="408" data-dominant-color="F8F3F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1400×829 22.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I get the message as below when I’m training WideResNet with CIFAR-10.</p>
<pre><code class="lang-auto">RuntimeError: Function 'LogSoftmaxBackward' returned nan values in its 0th output.
</code></pre>
<p>So I wonder what’s the problem and I found the input is having nan values using the codes below.</p>
<pre><code class="lang-auto">for epoch in range(epochs):
    for i, (X, y) in enumerate(train_data):
        X, y = X.cuda(self.gpu, non_blocking=True), y.cuda(self.gpu, non_blocking=True)
        for j in range(X.size(0)):
            if X[j].isnan().any():
                print(j, X[j])
</code></pre>
<p>But interesting thing is that if I index from dataset, it’s just fine (no nan value).</p>
<p>What’s the problem? I still cannot understand…</p>
<p>I’m using pytorch official docker (pytorch/pytorch:1.8.1-cuda11.1-cudnn8-devel) and GPU with A100-SXM4-40GB.</p>

username: Jason1995
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/e/8ea711317c381ef79a7d2dccf857ba10b38c56ec.png" data-download-href="https://discuss.pytorch.org/uploads/default/8ea711317c381ef79a7d2dccf857ba10b38c56ec" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/e/8ea711317c381ef79a7d2dccf857ba10b38c56ec.png" alt="image" data-base62-sha1="klXJUESmu1hZ9nifoQTKBT3xLQg" width="690" height="394" data-dominant-color="F8FAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1343×767 16.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre><code class="lang-auto">for epoch in range(epochs):
    for i, (X, y) in enumerate(train_data):
        print(X.isnan().any())
        X, y = X.cuda(self.gpu, non_blocking=True), y.cuda(self.gpu, non_blocking=True)
        print(X.isnan().any())
</code></pre>
<p>I checked more and just found that loading to gpu makes this weird behaviors.</p>
<p>Am I using wrong way to load tensors to gpu? Or it’s related with hardware issues (gpu)?</p>

username: ptrblck
cooked: <p>Try to synchronize via <code>torch.cuda.synchronize()</code> before using this tensor as you are using a non blocking data transfer.</p>

username: Jason1995
cooked: <p>It doesn’t work…<br>
Is there other way to prevent those weird behavior??<br>
btw I’m using single GPU (A100-SXM4-40GB)</p>

username: ptrblck
cooked: <p>Thanks for the update.<br>
Could you execute these runs and see, if the behavior changes, as a sync wouldn’t be strictly necessary, but would help us to isolate is an internal method is broken:</p>
<ul>
<li>add synchronizations in the <code>train_data</code> loop before and after each line of code</li>
<li>set <code>non_blocking=False</code>
</li>
<li>rerun the test with and without <code>pin_memory</code> in the <code>DataLoader</code>
</li>
</ul>
<p>Also, do you have a code snippet to reproduce the issue and could post the output of <code>python -m torch.utils.collect_env</code> here?</p>

username: Jason1995
cooked: <p>As you replied I tried and it worked!</p>
<p>If I set pin_memory=False, it works just fine <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>I share my experiment with ur suggestions as follows (u can see more details with codes)</p>
<ul>
<li>Experiment 1 - torch.cuda.synchronize() - not worked</li>
<li>Experiment 2 - non_blocking=False - not worked</li>
<li>Experiment 3 - pin_memory=False - worked</li>
</ul>
<p>Also I share my env for reproduction (for info, I used official pytorch docker image pytorch/pytorch:1.8.1-cuda11.1-cudnn8-devel and just install jupyter notebook using pip)</p>
<p>And just curious that I’m sharing 1 GPU with other users cause of lack of GPUs, is this matter…??</p>
<pre><code class="lang-auto"># env
!python -m torch.utils.collect_env 

# result
Collecting environment information...
PyTorch version: 1.8.1
Is debug build: False
CUDA used to build PyTorch: 11.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: Could not collect

Python version: 3.8 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: 
GPU 0: A100-SXM4-40GB
GPU 1: A100-SXM4-40GB
GPU 2: A100-SXM4-40GB
GPU 3: A100-SXM4-40GB
GPU 4: A100-SXM4-40GB
GPU 5: A100-SXM4-40GB
GPU 6: A100-SXM4-40GB
GPU 7: A100-SXM4-40GB

Nvidia driver version: 450.102.04
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.0.5
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.2
[pip3] torch==1.8.1
[pip3] torchelastic==0.2.2
[pip3] torchtext==0.9.1
[pip3] torchvision==0.9.1
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               11.1.74              h6bb024c_0    nvidia
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] mkl                       2020.2                      256  
[conda] mkl-service               2.3.0            py38he904b0f_0  
[conda] mkl_fft                   1.3.0            py38h54f3939_0  
[conda] mkl_random                1.1.1            py38h0573a6f_0  
[conda] numpy                     1.19.2           py38h54aff64_0  
[conda] numpy-base                1.19.2           py38hfa32c7d_0  
[conda] pytorch                   1.8.1           py3.8_cuda11.1_cudnn8.0.5_0    pytorch
[conda] torchelastic              0.2.2                    pypi_0    pypi
[conda] torchtext                 0.9.1                      py38    pytorch
[conda] torchvision               0.9.1                py38_cu111    pytorch
</code></pre>
<h2>
<a name="experiment-1-insert-torchcudasynchronize-didnt-work-1" class="anchor" href="#experiment-1-insert-torchcudasynchronize-didnt-work-1"></a>Experiment 1. Insert torch.cuda.synchronize() - didn’t work</h2>
<ul>
<li>I tried the experiments as below (adding more torch.cuda.synchronize())</li>
</ul>
<pre><code class="lang-auto"># Default Dataset, Dataloader codes for Experiment 1
batch_size = 128

train_transform = transforms.Compose([
    transforms.ToTensor(), transforms.Normalize(mean=[0.4921, 0.4828, 0.4474], std=[0.1950, 0.1922, 0.1940])])

test_transform = transforms.Compose([
    transforms.ToTensor(), transforms.Normalize(mean=[0.4921, 0.4828, 0.4474], std=[0.1950, 0.1922, 0.1940])])

train_dataset = torchvision.datasets.CIFAR10('./cifar10/', train=True, download=True, transform=train_transform)
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=batch_size, shuffle=False, pin_memory=True)

test_dataset = torchvision.datasets.CIFAR10('./cifar10/', train=False, download=True, transform=test_transform)
test_loader = torch.utils.data.DataLoader(test_dataset, batch_size=batch_size, shuffle=False, pin_memory=True)
</code></pre>
<h3>
<a name="experiment-1-1-2" class="anchor" href="#experiment-1-1-2"></a>Experiment 1-1</h3>
<pre><code class="lang-auto"># train code - 1
for epoch in range(epochs):
    if epoch % self.epoch_print == 0: print('Epoch {} Started...'.format(epoch+1))
    for i, (X, y) in enumerate(train_data):
        print(X.isnan().any())
        torch.cuda.synchronize(self.gpu)
        print(X.isnan().any())
        X, y = X.cuda(self.gpu, non_blocking=True), y.cuda(self.gpu, non_blocking=True)
        print(X.isnan().any())


# result
Epoch 1 Started...
tensor(False)
tensor(False)
tensor(True, device='cuda:1')
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-9-16c095588069&gt; in &lt;module&gt;
----&gt; 1 WRN_28_10.train(train_loader, test_loader, save, epochs, lr, momentum, weight_decay, nesterov, milestones)

/workspace/jaeseong_lee/WideResNet/train.py in train(self, train_data, test_data, save, epochs, lr, momentum, weight_decay, nesterov, milestones)
     46 
     47                 optimizer.zero_grad()
---&gt; 48                 loss.backward()
     49                 optimizer.step()
     50 

/opt/conda/lib/python3.8/site-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph, inputs)
    243                 create_graph=create_graph,
    244                 inputs=inputs)
--&gt; 245         torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
    246 
    247     def register_hook(self, hook):

/opt/conda/lib/python3.8/site-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)
    143         retain_graph = create_graph
    144 
--&gt; 145     Variable._execution_engine.run_backward(
    146         tensors, grad_tensors_, retain_graph, create_graph, inputs,
    147         allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag

RuntimeError: Function 'LogSoftmaxBackward' returned nan values in its 0th output.
</code></pre>
<h3>
<a name="experiment-1-2-3" class="anchor" href="#experiment-1-2-3"></a>Experiment 1-2</h3>
<pre><code class="lang-auto"># train code - 2
for epoch in range(epochs):
    if epoch % self.epoch_print == 0: print('Epoch {} Started...'.format(epoch+1))
    for i, (X, y) in enumerate(train_data):
        print(X.isnan().any())
        torch.cuda.synchronize(self.gpu)
        print(X.isnan().any())
        X, y = X.cuda(self.gpu, non_blocking=True), y.cuda(self.gpu, non_blocking=True)
        print(X.isnan().any())
        torch.cuda.synchronize(self.gpu)
        print(X.isnan().any())


# result
Epoch 1 Started...
tensor(False)
tensor(False)
tensor(True, device='cuda:1')
tensor(True, device='cuda:1')
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-9-16c095588069&gt; in &lt;module&gt;
----&gt; 1 WRN_28_10.train(train_loader, test_loader, save, epochs, lr, momentum, weight_decay, nesterov, milestones)

/workspace/jaeseong_lee/WideResNet/train.py in train(self, train_data, test_data, save, epochs, lr, momentum, weight_decay, nesterov, milestones)
     46 
     47                 optimizer.zero_grad()
---&gt; 48                 loss.backward()
     49                 optimizer.step()
     50 

/opt/conda/lib/python3.8/site-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph, inputs)
    243                 create_graph=create_graph,
    244                 inputs=inputs)
--&gt; 245         torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
    246 
    247     def register_hook(self, hook):

/opt/conda/lib/python3.8/site-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)
    143         retain_graph = create_graph
    144 
--&gt; 145     Variable._execution_engine.run_backward(
    146         tensors, grad_tensors_, retain_graph, create_graph, inputs,
    147         allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag

RuntimeError: Function 'LogSoftmaxBackward' returned nan values in its 0th output.
</code></pre>
<h3>
<a name="experiment-1-3-4" class="anchor" href="#experiment-1-3-4"></a>Experiment 1-3</h3>
<pre><code class="lang-auto"># train code - 3
for epoch in range(epochs):
    if epoch % self.epoch_print == 0: print('Epoch {} Started...'.format(epoch+1))
    torch.cuda.synchronize(self.gpu)
    for i, (X, y) in enumerate(train_data):
        print(X.isnan().any())
        torch.cuda.synchronize(self.gpu)
        print(X.isnan().any())
        X, y = X.cuda(self.gpu, non_blocking=True), y.cuda(self.gpu, non_blocking=True)
        print(X.isnan().any())
        torch.cuda.synchronize(self.gpu)
        print(X.isnan().any())


# result
Epoch 1 Started...
tensor(False)
tensor(False)
tensor(True, device='cuda:1')
tensor(True, device='cuda:1')
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-9-16c095588069&gt; in &lt;module&gt;
----&gt; 1 WRN_28_10.train(train_loader, test_loader, save, epochs, lr, momentum, weight_decay, nesterov, milestones)

/workspace/jaeseong_lee/WideResNet/train.py in train(self, train_data, test_data, save, epochs, lr, momentum, weight_decay, nesterov, milestones)
     46 
     47                 optimizer.zero_grad()
---&gt; 48                 loss.backward()
     49                 optimizer.step()
     50 

/opt/conda/lib/python3.8/site-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph, inputs)
    243                 create_graph=create_graph,
    244                 inputs=inputs)
--&gt; 245         torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
    246 
    247     def register_hook(self, hook):

/opt/conda/lib/python3.8/site-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)
    143         retain_graph = create_graph
    144 
--&gt; 145     Variable._execution_engine.run_backward(
    146         tensors, grad_tensors_, retain_graph, create_graph, inputs,
    147         allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag

RuntimeError: Function 'LogSoftmaxBackward' returned nan values in its 0th output.
</code></pre>
<h3>
<a name="experiment-1-4-5" class="anchor" href="#experiment-1-4-5"></a>Experiment 1-4</h3>
<pre><code class="lang-auto"># train code - 4
torch.cuda.synchronize(self.gpu)
for epoch in range(epochs):
    if epoch % self.epoch_print == 0: print('Epoch {} Started...'.format(epoch+1))
    torch.cuda.synchronize(self.gpu)
    for i, (X, y) in enumerate(train_data):
        print(X.isnan().any())
        torch.cuda.synchronize(self.gpu)
        print(X.isnan().any())
        X, y = X.cuda(self.gpu, non_blocking=True), y.cuda(self.gpu, non_blocking=True)
        print(X.isnan().any())
        torch.cuda.synchronize(self.gpu)
        print(X.isnan().any())


# result
Epoch 1 Started...
tensor(False)
tensor(False)
tensor(True, device='cuda:1')
tensor(True, device='cuda:1')
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-9-16c095588069&gt; in &lt;module&gt;
----&gt; 1 WRN_28_10.train(train_loader, test_loader, save, epochs, lr, momentum, weight_decay, nesterov, milestones)

/workspace/jaeseong_lee/WideResNet/train.py in train(self, train_data, test_data, save, epochs, lr, momentum, weight_decay, nesterov, milestones)
     46 
     47                 optimizer.zero_grad()
---&gt; 48                 loss.backward()
     49                 optimizer.step()
     50 

/opt/conda/lib/python3.8/site-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph, inputs)
    243                 create_graph=create_graph,
    244                 inputs=inputs)
--&gt; 245         torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
    246 
    247     def register_hook(self, hook):

/opt/conda/lib/python3.8/site-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)
    143         retain_graph = create_graph
    144 
--&gt; 145     Variable._execution_engine.run_backward(
    146         tensors, grad_tensors_, retain_graph, create_graph, inputs,
    147         allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag

RuntimeError: Function 'LogSoftmaxBackward' returned nan values in its 0th output.
</code></pre>
<h2>
<a name="experiment-2-set-non_blockingfalse-didnt-work-6" class="anchor" href="#experiment-2-set-non_blockingfalse-didnt-work-6"></a>Experiment 2. Set non_blocking=False - didn’t work</h2>
<pre><code class="lang-auto"># Default Dataset, Dataloader codes for Experiment 2
batch_size = 128

train_transform = transforms.Compose([
    transforms.ToTensor(), transforms.Normalize(mean=[0.4921, 0.4828, 0.4474], std=[0.1950, 0.1922, 0.1940])])

test_transform = transforms.Compose([
    transforms.ToTensor(), transforms.Normalize(mean=[0.4921, 0.4828, 0.4474], std=[0.1950, 0.1922, 0.1940])])

train_dataset = torchvision.datasets.CIFAR10('./cifar10/', train=True, download=True, transform=train_transform)
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=batch_size, shuffle=False, pin_memory=True)

test_dataset = torchvision.datasets.CIFAR10('./cifar10/', train=False, download=True, transform=test_transform)
test_loader = torch.utils.data.DataLoader(test_dataset, batch_size=batch_size, shuffle=False, pin_memory=True)
</code></pre>
<h3>
<a name="experiment-2-1-without-inserting-torchcudasynchronize-7" class="anchor" href="#experiment-2-1-without-inserting-torchcudasynchronize-7"></a>Experiment 2-1 (without inserting torch.cuda.synchronize())</h3>
<pre><code class="lang-auto"># train code - 1
#torch.cuda.synchronize(self.gpu)
for epoch in range(epochs):
    if epoch % self.epoch_print == 0: print('Epoch {} Started...'.format(epoch+1))
    #torch.cuda.synchronize(self.gpu)
    for i, (X, y) in enumerate(train_data):
        print(X.isnan().any())
        #torch.cuda.synchronize(self.gpu)
        print(X.isnan().any())
        X, y = X.cuda(self.gpu, non_blocking=False), y.cuda(self.gpu, non_blocking=False)
        print(X.isnan().any())
        #torch.cuda.synchronize(self.gpu)
        print(X.isnan().any())


# result
Epoch 1 Started...
tensor(False)
tensor(False)
tensor(True, device='cuda:1')
tensor(True, device='cuda:1')
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-9-16c095588069&gt; in &lt;module&gt;
----&gt; 1 WRN_28_10.train(train_loader, test_loader, save, epochs, lr, momentum, weight_decay, nesterov, milestones)

/workspace/jaeseong_lee/WideResNet/train.py in train(self, train_data, test_data, save, epochs, lr, momentum, weight_decay, nesterov, milestones)
     46 
     47                 optimizer.zero_grad()
---&gt; 48                 loss.backward()
     49                 optimizer.step()
     50 

/opt/conda/lib/python3.8/site-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph, inputs)
    243                 create_graph=create_graph,
    244                 inputs=inputs)
--&gt; 245         torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
    246 
    247     def register_hook(self, hook):

/opt/conda/lib/python3.8/site-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)
    143         retain_graph = create_graph
    144 
--&gt; 145     Variable._execution_engine.run_backward(
    146         tensors, grad_tensors_, retain_graph, create_graph, inputs,
    147         allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag

RuntimeError: Function 'LogSoftmaxBackward' returned nan values in its 0th output.
</code></pre>
<h3>
<a name="experiment-2-2-inserting-torchcudasynchronize-as-experiment-1-4-8" class="anchor" href="#experiment-2-2-inserting-torchcudasynchronize-as-experiment-1-4-8"></a>Experiment 2-2 (inserting torch.cuda.synchronize() as Experiment 1-4)</h3>
<pre><code class="lang-auto"># train code - 2
torch.cuda.synchronize(self.gpu)
for epoch in range(epochs):
    if epoch % self.epoch_print == 0: print('Epoch {} Started...'.format(epoch+1))
    torch.cuda.synchronize(self.gpu)
    for i, (X, y) in enumerate(train_data):
        print(X.isnan().any())
        torch.cuda.synchronize(self.gpu)
        print(X.isnan().any())
        X, y = X.cuda(self.gpu, non_blocking=False), y.cuda(self.gpu, non_blocking=False)
        print(X.isnan().any())
        torch.cuda.synchronize(self.gpu)
        print(X.isnan().any())


# result
Epoch 1 Started...
tensor(False)
tensor(False)
tensor(True, device='cuda:1')
tensor(True, device='cuda:1')
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-9-16c095588069&gt; in &lt;module&gt;
----&gt; 1 WRN_28_10.train(train_loader, test_loader, save, epochs, lr, momentum, weight_decay, nesterov, milestones)

/workspace/jaeseong_lee/WideResNet/train.py in train(self, train_data, test_data, save, epochs, lr, momentum, weight_decay, nesterov, milestones)
     46 
     47                 optimizer.zero_grad()
---&gt; 48                 loss.backward()
     49                 optimizer.step()
     50 

/opt/conda/lib/python3.8/site-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph, inputs)
    243                 create_graph=create_graph,
    244                 inputs=inputs)
--&gt; 245         torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
    246 
    247     def register_hook(self, hook):

/opt/conda/lib/python3.8/site-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)
    143         retain_graph = create_graph
    144 
--&gt; 145     Variable._execution_engine.run_backward(
    146         tensors, grad_tensors_, retain_graph, create_graph, inputs,
    147         allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag

RuntimeError: Function 'LogSoftmaxBackward' returned nan values in its 0th output.
</code></pre>
<h2>
<a name="experiment-3-pin-memoryfalse-works-well-9" class="anchor" href="#experiment-3-pin-memoryfalse-works-well-9"></a>Experiment 3. pin memory=False - works well</h2>
<h3>
<a name="experiment-3-1-without-torchcudasynchronize-non_blockingtrue-10" class="anchor" href="#experiment-3-1-without-torchcudasynchronize-non_blockingtrue-10"></a>Experiment 3-1 (without torch.cuda.synchronize(), non_blocking=True)</h3>
<pre><code class="lang-auto"># Dataset, Dataloader codes
batch_size = 128

train_transform = transforms.Compose([
    transforms.ToTensor(), transforms.Normalize(mean=[0.4921, 0.4828, 0.4474], std=[0.1950, 0.1922, 0.1940])])

test_transform = transforms.Compose([
    transforms.ToTensor(), transforms.Normalize(mean=[0.4921, 0.4828, 0.4474], std=[0.1950, 0.1922, 0.1940])])

train_dataset = torchvision.datasets.CIFAR10('./cifar10/', train=True, download=True, transform=train_transform)
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=batch_size, shuffle=False, pin_memory=False)

test_dataset = torchvision.datasets.CIFAR10('./cifar10/', train=False, download=True, transform=test_transform)
test_loader = torch.utils.data.DataLoader(test_dataset, batch_size=batch_size, shuffle=False, pin_memory=False)


# train code - 1
#torch.cuda.synchronize(self.gpu)
for epoch in range(epochs):
    if epoch % self.epoch_print == 0: print('Epoch {} Started...'.format(epoch+1))
    #torch.cuda.synchronize(self.gpu)
    for i, (X, y) in enumerate(train_data):
        print(X.isnan().any())
        #torch.cuda.synchronize(self.gpu)
        print(X.isnan().any())
        X, y = X.cuda(self.gpu, non_blocking=True), y.cuda(self.gpu, non_blocking=True)
        print(X.isnan().any())
        #torch.cuda.synchronize(self.gpu)
        print(X.isnan().any())


# result
Epoch 1 Started...
tensor(False)
tensor(False)
tensor(False, device='cuda:1')
tensor(False, device='cuda:1')
tensor(False)
tensor(False)
tensor(False, device='cuda:1')
tensor(False, device='cuda:1')
tensor(False)
tensor(False)
tensor(False, device='cuda:1')
tensor(False, device='cuda:1')
tensor(False)
tensor(False)
tensor(False, device='cuda:1')
tensor(False, device='cuda:1')
</code></pre>

username: ptrblck
cooked: <p>Thanks for the detailed analysis as well as the minimal code snippet, which is really helpful!<br>
I’ll try to reproduce it using the wheels on our systems.</p>

username: ptrblck
cooked: <p>Unfortunately, I cannot reproduce the issue on our 8x A100 node using the pip wheels, conda binaries, and a source build. Are you seeing the same behavior on all GPUs (could you try to use another one, as you are currently using <code>GPU1</code>)? Did this behavior start after a specific update (of the driver etc.)?</p>

username: Jason1995
cooked: <h2><a name="h-1-checking-on-other-gpus-1" class="anchor" href="#h-1-checking-on-other-gpus-1"></a>1. Checking on other gpus</h2>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/2/322d6ebb59be8a736a578acbeb4661d61425cf7c.png" data-download-href="https://discuss.pytorch.org/uploads/default/322d6ebb59be8a736a578acbeb4661d61425cf7c" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/2/322d6ebb59be8a736a578acbeb4661d61425cf7c.png" alt="image" data-base62-sha1="79Tda5X6TfnCOHFjlKtStxYSg8s" width="159" height="500" data-dominant-color="F8F8F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">371×1163 7.25 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Since I found this error on sunday, I mostly checked on the gpus I can access on sunday.<br>
(This is the capture report I wrote for bugs issues to send which I tried on sunday, checked on gpu - 0, 1, 3, 4, 5, 6, 7)</p>
<p>It’s some kinda weird that sometimes it’s okay but sometimes it’s making weird behaviors.<br>
(See gpu 0 and 6 - sometimes it’s okay but sometimes it makes nan)</p>
<p>And still I can find the same issues on gpu - 0, 1, 3, 4, 6<br>
(cause I’m using gpu server with other users, it’s quite hard to check on all gpus… but I checked as much as I can)</p>
<p>Also, some trials go well but some trials make nan behaviors.</p>
<p>For more details, I upload my debug codes on google drive links with open access!</p>
<p>I run my codes using jupyter so u can check the logs that I printed.</p>
<p><a href="https://drive.google.com/drive/folders/1eQavapn1NQ_NQkU6Kg84WX42UYy7CnSD" class="onebox" target="_blank" rel="noopener nofollow ugc">https://drive.google.com/drive/folders/1eQavapn1NQ_NQkU6Kg84WX42UYy7CnSD</a></p>
<h2><a name="h-2-specific-updates-2" class="anchor" href="#h-2-specific-updates-2"></a>2. Specific updates</h2>
<p>Actually I’m using gpu server as rent, I don’t know the owner did some special updates or not…</p>
<p>I know it could be hardware issues and also the software issues so I asked for gpu server owner for this issues.</p>
<p>Also I’m not the only one who is using gpu server, that might be the reason?? (like there are more than 2 processes are working per 1 gpu for every 8 gpus)</p>
<p>If you need more details, I’ll try to reproduce it or explain I can do</p>

username: ptrblck
cooked: <p>Multiple processes using the same devices shouldn’t cause this issue and since I cannot reproduce the issue on our A100 servers, I would guess it might be a setup issue. Hardware defects could of course also cause such weird issues, but since you are able to reproduce it on multiple devices in the server, I doubt it’s the case.<br>
Did the admin provide any information about the setup of the machine? I’m also not familiar with your lease of the node, but could you try another one?</p>

username: Jason1995
cooked: <p>I don’t have more available server to check <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>
<p>I send emails to admin (the server owner) and if I get replies, I’ll reply for u!</p>

username: jopo666
cooked: <p>I’ve been having the same problem for the past 6 months! Pain in the ass to reproduce as it only happens every now and then. On CPU the tensors are fine but when sending them to GPU, some individual values may flip to <code>nan</code>.</p>
<p>I’ve never observed the problem in our 4 x 4 P100 server if I run my models on a single node but when training models on multiple nodes (SLURM and DPP) this problem occurs again. I’ve had talks about this problem with out server provider and we tried swapping GPUs but the problem persists.</p>
<p><code>pin_memory=False</code> seems to not fix the problem for me. I’m happy to provide additional details about our system if it helps! Here’s the output of <code>collect_env</code> if it helps at all.</p>
<pre><code class="lang-auto">Collecting environment information...
PyTorch version: 1.8.1+cu102
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.2 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: Could not collect
CMake version: Could not collect

Python version: 3.8 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: 
GPU 0: Tesla P100-PCIE-16GB
GPU 1: Tesla P100-PCIE-16GB
GPU 2: Tesla P100-PCIE-16GB
GPU 3: Tesla P100-PCIE-16GB

Nvidia driver version: 450.102.04
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.20.2
[pip3] pytorch-lightning==1.3.0
[pip3] torch==1.8.1
[pip3] torchmetrics==0.2.0
[pip3] torchvision==0.9.1
[conda] _pytorch_select           0.1                       cpu_0  
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               11.0.221             h6bb024c_0  
[conda] libmklml                  2019.0.5                      0  
[conda] mkl                       2020.2                      256  
[conda] mkl-service               2.3.0            py38he904b0f_0  
[conda] mkl_fft                   1.3.0            py38h54f3939_0  
[conda] mkl_random                1.1.1            py38h0573a6f_0  
[conda] numpy                     1.20.2                   pypi_0    pypi
[conda] numpy-base                1.19.2           py38hfa32c7d_0  
[conda] pytorch-lightning         1.3.0                    pypi_0    pypi
[conda] torch                     1.8.1                    pypi_0    pypi
[conda] torchmetrics              0.2.0                    pypi_0    pypi
[conda] torchvision               0.9.1                    pypi_0    pypi
</code></pre>