Issue 1:
🐛 Bug

We have a number of machines with Threadripper CPUs, and 2 NVIDIA GPUs, some have 1070ti cards some 1080 some 1080ti and one with titanXp, they all displayed this behavior, when switching to using data parallel, training would fail, i.e. accuracy would not go up. We first saw this in our code base, but it also happens on the imagnet example from the pytorch examples repo

To Reproduce

Steps to reproduce the behavior:

run the imagnet example for the examples repo in pytorch with dataparallel

these error messages were found in the dmesg log:

[1118468.873266] nvidia 0000:0a:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x000f address=0x00000000ea13a000 flags=0x0020]

[1118468.942145] nvidia 0000:0a:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x000f address=0x00000000ea139068 flags=0x0020]

[1118468.942189] nvidia 0000:0a:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x000f address=0x00000000d0000040 flags=0x0020]

[1118468.942227] nvidia 0000:0a:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x000f address=0x00000000d00007c0 flags=0x0020]

[1118468.942265] nvidia 0000:0a:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x000f address=0x00000000d0001040 flags=0x0020]

[1118468.942303] nvidia 0000:0a:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x000f address=0x00000000d0000f40 flags=0x0020]

[1118468.942340] nvidia 0000:0a:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x000f address=0x00000000d00016c0 flags=0x0020]

[1118468.942377] nvidia 0000:0a:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x000f address=0x00000000d0002040 flags=0x0020]

[1118468.942414] nvidia 0000:0a:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x000f address=0x00000000d0001e40 flags=0x0020]

[1118468.942452] nvidia 0000:0a:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x000f address=0x00000000d00025c0 flags=0x0020]

[1118468.942489] AMD-Vi: Event logged [IO_PAGE_FAULT device=0a:00.0 domain=0x000f address=0x00000000d0003040 flags=0x0020]

[1118468.942525] AMD-Vi: Event logged [IO_PAGE_FAULT device=0a:00.0 domain=0x000f address=0x00000000d0002d40 flags=0x0020]

[1118468.942560] AMD-Vi: Event logged [IO_PAGE_FAULT device=0a:00.0 domain=0x000f address=0x00000000d00034c0 flags=0x0020]

[1118468.942596] AMD-Vi: Event logged [IO_PAGE_FAULT device=0a:00.0 domain=0x000f address=0x00000000d0004040 flags=0x0020]

[1118468.942632] AMD-Vi: Event logged [IO_PAGE_FAULT device=0a:00.0 domain=0x000f address=0x00000000d0003c40 flags=0x0020]

[1118468.942667] AMD-Vi: Event logged [IO_PAGE_FAULT device=0a:00.0 domain=0x000f address=0x00000000d00043c0 flags=0x0020]

[1118468.942703] AMD-Vi: Event logged [IO_PAGE_FAULT device=0a:00.0 domain=0x000f address=0x00000000d0005040 flags=0x0020]

[1118468.942739] AMD-Vi: Event logged [IO_PAGE_FAULT device=0a:00.0 domain=0x000f address=0x00000000d0004b40 flags=0x0020]

[1118468.942774] AMD-Vi: Event logged [IO_PAGE_FAULT device=0a:00.0 domain=0x000f address=0x00000000d00052c0 flags=0x0020]

Expected behavior

accuracy would not pick up in most cases, it never picked up for the validation set. We managed to work around this problem by turning off IOMMU in the bios.

Environment

PyTorch version: 0.4.1

Is debug build: No

CUDA used to build PyTorch: 9.0.176

OS: Ubuntu 16.04.5 LTS

GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609

CMake version: version 3.5.1

Python version: 2.7

Is CUDA available: Yes

CUDA runtime version: Could not collect

GPU models and configuration:

GPU 0: GeForce GTX 1080 Ti

GPU 1: GeForce GTX 1080 Ti

Nvidia driver version: 384.130

cuDNN version: Probably one of the following:

/usr/local/cuda-9.0/targets/x86_64-linux/lib/libcudnn.so.7.0.5

/usr/local/cuda-9.0/targets/x86_64-linux/lib/libcudnn_static.a

Issue 2:
I'm having trouble getting multi-gpu via DataParallel across two Tesla K80 GPUs. The code I'm using is a modification of the MNIST example:

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torchvision import datasets, transforms
from torch.autograd import Variable
from data_parallel import DataParallel

train_loader = torch.utils.data.DataLoader(
datasets.MNIST('../data', train=True, download=True,
transform=transforms.Compose([
transforms.ToTensor(),
transforms.Normalize((0.1307,), (0.3081,))
])),
batch_size=256, shuffle=True, num_workers=2, pin_memory=True)

class Net(nn.Module):
def __init__(self):
super(Net, self).__init__()
self.conv1 = nn.Conv2d(1, 10, kernel_size=5)
self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
self.fc1 = nn.Linear(320, 50)
self.fc2 = nn.Linear(50, 10)

def forward(self, x):
x = F.relu(F.max_pool2d(self.conv1(x), 2))
x = F.relu(F.max_pool2d(self.conv2(x), 2))
x = x.view(-1, 320)
x = F.relu(self.fc1(x))
x = self.fc2(x)
return F.log_softmax(x)

model = DataParallel(Net())
model.cuda()

optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.9)
criterion = nn.NLLLoss().cuda()

model.train()
for batch_idx, (data, target) in enumerate(train_loader):
input_var = Variable(data.cuda())
target_var = Variable(target.cuda())

print('Getting model output')
output = model(input_var)
print('Got model output')

loss = criterion(output, target_var)
optimizer.zero_grad()
loss.backward()
optimizer.step()

print('Finished')

This doesn't throw an error, but hangs after it prints "Getting model output" and never returns. I traced this down to the parallel_apply spawning threads that then never finish. The line that hangs is here where the threads are spawned using both GPU 0 and GPU 1, but never finish.

This is only a problem when CUDA_VISIBLE_DEVICES=0,1 as both GPU0 and GPU1 work perfectly well individually.

Before running this, nvidia-smi shows

+------------------------------------------------------+
| NVIDIA-SMI 352.68 Driver Version: 352.68 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K80 Off | 0000:06:00.0 Off | 0 |
| N/A 40C P0 57W / 149W | 55MiB / 11519MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K80 Off | 0000:07:00.0 Off | 0 |
| N/A 35C P0 76W / 149W | 55MiB / 11519MiB | 99% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+

after running (while it hangs), nvidia-smi gives

+------------------------------------------------------+
| NVIDIA-SMI 352.68 Driver Version: 352.68 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K80 Off | 0000:06:00.0 Off | 0 |
| N/A 42C P0 69W / 149W | 251MiB / 11519MiB | 99% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K80 Off | 0000:07:00.0 Off | 0 |
| N/A 36C P0 90W / 149W | 249MiB / 11519MiB | 99% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 4785 C python 194MiB |
| 1 4785 C python 192MiB |
+-----------------------------------------------------------------------------+

and top shows the main python process and the two python subprocesses. Wondering if this could be something similar to #554.

Using this TensorFlow example, I get linear speedup using multiple GPUs as I change CUDA_VISIBLE_DEVICES so multiple K80s should certainly be viable.