Errors occurred when using multiprocessing

username: Audrey
cooked: <p>Hi! I want to use <code>torch.multiuprocessing</code> to speed-up my training process. In short, the original training structure is as below.</p>
<pre><code class="lang-python">train_loader = DataLoader(train_dataset, batch_size=train_batch, shuffle=True)
model = Model(...)
optimizer = optim.SGD(model.parameters(), ...)
for i in range(epochs):
    for pos, neg in enumerate(train_loader):
        pos = pos.to(device).to(torch.long)
        neg = neg.to(device).to(torch.long)
        optimizer.zero_grad()
        loss = model(pos, neg)
        loss.backward()
        optimizer.step()
</code></pre>
<p>I want to use multiprocessing in each epoch, so I changed my code structure into this:</p>
<pre><code class="lang-python">def train(model, train_loder, device):
    optimizer = optim.SGD(model.parameters(), ...)
    for pos, neg in enumerate(train_loader):
        pos = pos.to(device).to(torch.long)
        neg = neg.to(device).to(torch.long)
        optimizer.zero_grad()
        loss = model(pos, neg)
        loss.backward()
        optimizer.step()

train_loader = DataLoader(train_dataset, batch_size=train_batch, shuffle=True)
model = Model(...)
for i in range(epochs):
    processes = []
    for rank in range(num_processes):
        p = mp.Process(target=train, args=(model, train_loader, device))
        p.start()
        processes.append(p)
    for p in processes:
        p.join()
</code></pre>
<p>But when I ran, I got errors I don’t understand: (<strong>My system is Win10</strong>)</p>
<pre><code class="lang-auto">THCudaCheck FAIL file=C:\w\1\s\windows\pytorch\torch/csrc/generic/StorageSharing.cpp line=245 error=71 : operation not supported
Traceback (most recent call last):
  File "D:/PyCharm/Projects/TransE/train.py", line 113, in &lt;module&gt;
    p.start()
  File "D:\Python3.7.3\lib\multiprocessing\process.py", line 112, in start
    self._popen = self._Popen(self)
  File "D:\Python3.7.3\lib\multiprocessing\context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "D:\Python3.7.3\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "D:\Python3.7.3\lib\multiprocessing\popen_spawn_win32.py", line 89, in __init__
    reduction.dump(process_obj, to_child)
  File "D:\Python3.7.3\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "D:\Python3.7.3\lib\site-packages\torch\multiprocessing\reductions.py", line 232, in reduce_tensor
    event_sync_required) = storage._share_cuda_()
RuntimeError: cuda runtime error (71) : operation not supported at C:\w\1\s\windows\pytorch\torch/csrc/generic/StorageSharing.cpp:245
Traceback (most recent call last):
  File "&lt;string&gt;", line 1, in &lt;module&gt;
  File "D:\Python3.7.3\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "D:\Python3.7.3\lib\multiprocessing\spawn.py", line 115, in _main
    self = reduction.pickle.load(from_parent)
</code></pre>
<p>Is my structure wrong? And how to fix it? Thanks!</p>
<p>PS: <em>It seems that there are not many tutorials on how to use torch.multiprocessing. And I am not familiar with multiprocessing</em> <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"> …</p>

username: albanD
cooked: <p>Hi,</p>
<p>the error states that <code>THCudaCheck</code>, in <code>StorageSharing.cpp</code> you are doing <code>operation not supported</code>.<br>
So my guess is that sharing cuda Tensors is not supported on windows.</p>
<p>Anyway, in general, multiprocessing is very unlikely to give you any performance benefit. What are the properties of your workload? Why do you want to use multiprocessing?</p>

username: Audrey
cooked: <p>Hi! Thanks for your answer. In my current situation, without multiprocessing each epoch costs about 8 seconds. I saw other implementations used <code>multiprocessing</code> to speed-up training process and it really helped (but they are written with Tensorflow).<br>
You said it’s likely that sharing cuda Tensors is not supported on windows. To verify this I tried runing a PyTorch official example program (<a href="https://github.com/pytorch/examples/blob/master/mnist_hogwild" rel="nofollow noopener">code link</a>) which uses a very similar multiprocessing structure as mine, and same error occurred if I used cuda in windows system. I believe your guess in right!</p>
<p>But I still want to speed-up my process. In the next try, I will use <code>multiprocessing</code> not <code>torch.multiprocessing</code> and see if it works!</p>