Issue 1:
Reported at https://discuss.pytorch.org/t/torch-multiprocessing-rebuild-cuda-tensor-having-trouble-with-bn-num-batches-tracked/27453

Post copied below:

I am trying to pass a set of gpu models to a pool of processes, so something like the example code below:

(torch version 1.0.0.dev20181015)

import torch.multiprocessing as mp
mp.set_start_method('spawn', force=True)
import torch
import torch.nn as nn
import time

def run(inpt):
workers, worker_queue = inpt
while worker_queue.empty():
time.sleep(0.1)
worker_id = worker_queue.get()
model = workers[worker_id]
data = torch.rand(5, 10)
data = data.cuda()
with torch.no_grad():
result = model(data)
worker_queue.put(worker_id)
return result

def main():
workers = []
worker_queue = mp.Manager().Queue() # mp.Queue()
for worker_id in range(10):
worker_model = nn.BatchNorm1d(10)
worker_model.cuda()
worker_model.eval()
workers.append(worker_model)
worker_queue.put(worker_id)

pools = mp.Pool(10)

jobs = [(workers, worker_queue) for _ in range(20)] # jobs
result = []
for res_id, res in enumerate(pools.imap_unordered(run, jobs)):
result.append(res)
print(result)

if __name__=="__main__":
main()

However, when there is BatchNorm, the following error is raised in torch._utils.rebuild_cuda_tensor:

RuntimeError: Expected object of data type 4 but got data type 6 for argument #2 'source'

It is probably caused by the attribute bn.num_batches_tracked, which is supposed to be type 4 (int64_t), but looks like the error message is reporting data type 6 (float). What is likely the cause of this issue? Is there any way to fix it?

Issue 2:
I was trying to send multiple tensors in tuple via multiprocessing.Queue to anther process. However when I access to a cuda LongTensor, I get this error

RuntimeError: Attempt to access Storage having data type Float as data type Long

Code example

import torch
import torch.multiprocessing as mp
import time

def producer(queue):
while True:
a = torch.ones(2,2).float().cuda()
idx = torch.LongTensor([[0, 0], [0, 1]]).cuda()
queue.put((a, idx))

def consumer(queue):
while True:
a, idx = queue.get()
print(idx.type())
print(idx)

if __name__ == '__main__':
mp.set_start_method('spawn')

queue = mp.Queue()

p = mp.Process(target=producer, args=(queue,))
c = mp.Process(target=consumer, args=(queue,))
p.start()
c.start()

time.sleep(10)

p.join()
c.join()

Output:

torch.cuda.LongTensor
Process Process-2:
Traceback (most recent call last):
File "***/lib/python3.5/multiprocessing/process.py", line 252, in _bootstrap
self.run()
File "***/lib/python3.5/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "queue_test.py", line 15, in consumer
print(idx)
File "***/lib/python3.5/site-packages/torch/tensor.py", line 57, in __repr__
return torch._tensor_str._str(self)
File "***/lib/python3.5/site-packages/torch/_tensor_str.py", line 256, in _str
formatter = _Formatter(get_summarized_data(self) if summarize else self)
File "***/lib/python3.5/site-packages/torch/_tensor_str.py", line 76, in __init__
copy = torch.empty(tensor.size(), dtype=torch.long).copy_(tensor).view(tensor.nelement())
RuntimeError: Attempt to access Storage having data type Float as data type Long

System Info

PyTorch version: 0.4.1

Is debug build: No

CUDA used to build PyTorch: 8.0.61

OS: Ubuntu 14.04.5 LTS

GCC version: (Ubuntu 4.8.4-2ubuntu1~14.04.4) 4.8.4

CMake version: version 3.11.0

Python version: 3.5

Is CUDA available: Yes

CUDA runtime version: 8.0.61

GPU models and configuration:

GPU 0: Tesla P100-PCIE-16GB

GPU 1: Tesla P100-PCIE-16GB

Nvidia driver version: 375.66

cuDNN version: Probably one of the following:

/usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudnn.so

/usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudnn.so.5.1.10

/usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudnn.so.6

/usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudnn.so.6.0.21

/usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudnn_static.a

Versions of relevant libraries:

[pip] numpy (1.13.3)

[pip] torch (0.4.1)

[pip] torchvision (0.2.1)

[conda] torch 0.4.1

[conda] torchvision 0.2.1