Dataparallel tutorial and Cublas errors

Aaron_De_Rybel: <p>I’m trying to run the dataparallel tutorial on my own hardware<br>
<a href="https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html</a></p>
<p>I copy-pasted the code inside a script (see below) and run it through a terminal using<br>
<strong>CUDA_LAUNCH_BLOCKING=1 python dataparallel.py</strong></p>
<p>I get the following full stack trace, might be something wrong with my setup?<br>
I’ve installed torch on linux with simple pip install torch (version 1.8.0)<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/f/0f727e085af97bbaef401f8c0c0950e33d666f29.png" data-download-href="https://discuss.pytorch.org/uploads/default/0f727e085af97bbaef401f8c0c0950e33d666f29" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/f/0f727e085af97bbaef401f8c0c0950e33d666f29.png" alt="image" data-base62-sha1="2cEsyYaeiPw3CkvXKVrwuOPq8Pf" width="690" height="275" data-dominant-color="0A0B0B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1505×600 37.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
from torch.utils.data import Dataset, DataLoader

# Parameters and DataLoaders
input_size = 30
output_size = 2

batch_size = 30
data_size = 100

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

class RandomDataset(Dataset):

    def __init__(self, size, length):
        self.len = length
        self.data = torch.randn(length, size)

    def __getitem__(self, index):
        return self.data[index]

    def __len__(self):
        return self.len

rand_loader = DataLoader(dataset=RandomDataset(input_size, data_size),
                         batch_size=batch_size, shuffle=True)

class Model(nn.Module):
    # Our model

    def __init__(self, input_size, output_size):
        super(Model, self).__init__()
        self.fc = nn.Linear(input_size, output_size)

    def forward(self, input):
        output = self.fc(input)
        print("\tIn Model: input size", input.size(),
              "output size", output.size())

        return output

model = Model(input_size, output_size)
if torch.cuda.device_count() &gt; 1:
  print("Let's use", torch.cuda.device_count(), "GPUs!")
  # dim = 0 [30, xxx] -&gt; [10, ...], [10, ...], [10, ...] on 3 GPUs
  model = nn.DataParallel(model)

model.to(device)

for data in rand_loader:
    input = data.to(device)
    output = model(input)
    print("Outside: input size", input.size(),
          "output_size", output.size())
</code></pre>

aguennecjacq: <p>Can you run it block by block in order to understand where things are going wrong ?</p>

ptrblck: <p>Could you post the output of <code>python -m torch.utils.collect_env</code>?<br>
If you are using a Turing GPU and installed the PyTorch <code>1.8.0</code> pip wheels with the CUDA10.2 runtime, please refer to <a href="https://discuss.pytorch.org/t/cuda-error-cublas-status-internal-error-when-calling-cublascreate-handle/114341/9">this post</a> and either install a conda binary, the CUDA11.1 pip wheel, or any nightly release.</p>