Issue 1:
Issue description

nn.DataParallel uses memory on GPU 0 even if it's explicitly instructed to use 2 and 3.

Code example

No line in this code snippet touches cuda:0.

import os
import torch
import torch.nn as nn

class MM(nn.Module):

def __init__(self):
super().__init__()
self.block1 = nn.Linear(10, 20)

# wrap block2 in DataParallel
self.block2 = nn.Linear(20, 20)
self.block2 = nn.DataParallel(self.block2, device_ids=[3, 2])

self.block3 = nn.Linear(20, 20)

def forward(self, x):
x = self.block1(x)
x = self.block2(x)
x = self.block3(x)
return x

dev='cuda:3'

x=torch.zeros((64, 10), device=dev)
mm=MM()
mm.to(device=dev)

y=mm(x)
os.system('nvidia-smi')

However, nvidia-smi output indicates that GPU 0 is allocated memory

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 2407 G /usr/lib/xorg/Xorg 14MiB |
| 0 5848 C python 306MiB |
| 2 5848 C python 324MiB |
| 3 5848 C python 324MiB |
+-----------------------------------------------------------------------------+

System Info

PyTorch version: 0.4.0

Is debug build: No

CUDA used to build PyTorch: 8.0.61

OS: Ubuntu 16.04.2 LTS

GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.9) 5.4.0 20160609

CMake version: version 3.5.1

Python version: 3.6

Is CUDA available: Yes

CUDA runtime version: 8.0.61

GPU models and configuration:

GPU 0: Tesla K80

GPU 1: Tesla K80

GPU 2: Tesla K80

GPU 3: Tesla K80

Nvidia driver version: 384.111

cuDNN version: Probably one of the following:

/usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudnn.so

/usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudnn.so.5

/usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudnn.so.5.1.10

/usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudnn_static.a

Versions of relevant libraries:

[pip3] msgpack-numpy (0.4.2)

[pip3] numpy (1.13.3)

[pip3] numpydoc (0.6.0)

[pip3] torch (0.4.0)

[pip3] torchvision (0.2.1)

[pip3] torchx (0.0.1, /home/jimfan/code/torchx)

[conda] cuda80 1.0 0 soumith

[conda] pytorch 0.4.0 py36_cuda8.0.61_cudnn7.1.2_1 pytorch

[conda] torchvision 0.2.1 py36_1 pytorch

[conda] torchx 0.0.1

Issue 2:
Issue description

on a multi GPU system, if a GPU != 0 is used, pytorch will still allocate some memory on the GPU 0 - see the nvidia-smi screenshot below.

Code example

import torch
import torch.nn as nn
from torch.autograd import Variable
import torch.optim as optim........

torch.backends.cudnn.benchmark = True

class TestNet(nn.Module):
def __init__(self):
super(TestNet, self).__init__()
self.net1 = nn.Linear(100, 200)
self.net2 = nn.Linear(200, 1)
self.bn = nn.BatchNorm1d(200)
self.sigmoid = nn.Sigmoid()
self.ReLU = nn.ReLU(inplace=False)
self.drop = nn.Dropout(0.5)
...............
def forward(self, V):
return self.sigmoid(self.net2(self.drop(self.ReLU(self.bn(self.net1(V)))))).squeeze().

GPU = 1
net = TestNet()
criterion = nn.BCELoss()
net.cuda(GPU)
criterion.cuda(GPU)
V = Variable(torch.randn(100, 100)).cuda(GPU)
label = Variable(torch.randn(100)).cuda(GPU)
optim_betas = (0.9, 0.999)
optimizer = optim.Adam(filter(lambda p: p.requires_grad, net.parameters()), lr=0.01, betas=optim_betas)

net.train()
for i in range(0,1000000):
net.zero_grad()
r = net(V)
err = criterion(r, label)
err.backward()...

tester@NEWTITAN:~$ nvidia-smi

Sat Apr 28 23:45:03 2018

+-----------------------------------------------------------------------------+

| NVIDIA-SMI 387.26 Driver Version: 387.26 |

|-------------------------------+----------------------+----------------------+

| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |

| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |

|===============================+======================+======================|

| 0 TITAN X (Pascal) Off | 00000000:05:00.0 Off | N/A |

| 23% 33C P2 57W / 250W | 401MiB / 12189MiB | 0% Default |

+-------------------------------+----------------------+----------------------+

| 1 GeForce GTX 108... Off | 00000000:06:00.0 Off | N/A |

| 29% 33C P2 58W / 250W | 549MiB / 11172MiB | 6% Default |

+-------------------------------+----------------------+----------------------+

| 2 GeForce GTX 108... Off | 00000000:09:00.0 Off | N/A |

| 29% 36C P8 10W / 250W | 10MiB / 11172MiB | 0% Default |

+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+

| Processes: GPU Memory |

| GPU PID Type Process name Usage |

|=============================================================================|

| 0 1262 C python 391MiB |

| 1 1262 C python 539MiB |

+-----------------------------------------------------------------------------+