Multiple networks running in parallel on different CPUs

Petros_Christodoulou: <p>I’m trying to have different neural networks run in parallel on different CPUs but am finding that it isn’t leading to any sort of speed up compared to running them sequentially.</p>
<p>Below is my code that replicates the issue exactly. If you run this code it shows that with 2 processes it takes roughly twice as long as running it with 1 process but really it should take the same amount of time.</p>
<pre><code class="lang-auto">import time
import torch.multiprocessing as mp
import gym
import numpy as np
import copy
import torch.nn as nn
import torch

class NN(nn.Module):
    def __init__(self, output_dim):
        nn.Module.__init__(self)
        self.fc1 = nn.Linear(4, 50)
        self.fc2 = nn.Linear(50, 500)
        self.fc3 = nn.Linear(500, 5000)
        self.fc4 = nn.Linear(5000, output_dim)
        self.relu = nn.ReLU()
    
    def forward(self, x):
        x = self.relu(self.fc1(x))
        x = self.relu(self.fc2(x))
        x = self.relu(self.fc3(x))
        x = self.fc4(x)
        return x

def Worker(ix):
  print("Starting training for worker ", ix)
  env = gym.make('CartPole-v0')
  model = NN(2)
  for _ in range(2000):
    model(torch.Tensor(env.reset()))
  print("Finishing training for worker ", ix)

def overall_process(num_workers):
  workers = []
  for ix in range(num_workers):
    worker = mp.Process(target=Worker, args=(ix, ))
    workers.append(worker)
  [w.start() for w in workers]  
  for worker in workers:
    worker.join()

  print("Finished Training")  
  print(" ")

start = time.time()
overall_process(1)
print("Time taken: ", time.time() - start)
print(" ")

start = time.time()
overall_process(2)
print("Time taken: ", time.time() - start)
</code></pre>
<p>Does anyone know why this might be happening and how to fix it?</p>
<p>I thought that it is maybe because PyTorch networks automatically implement CPU parallelism in the background and so I tried adding the below 2 lines but it doesn’t always resolve the issue:</p>
<pre><code class="lang-auto">torch.set_num_threads(1)
torch.set_num_interop_threads(1)
</code></pre>

mrshenli: <p>Hey <a class="mention" href="/u/vitalyfedyunin">@VitalyFedyunin</a> <a class="mention" href="/u/alband">@albanD</a> , do you know what could cause the seemingly sequential execution?</p>
<p>I tried the script on my laptop and saw the same behavior. When I increase the # of iterations from 2000 to 20000, it becomes even worse.</p>
<p>1 process<br>
Time taken:  72.73709082603455</p>
<p>2 processes<br>
Time taken:  229.3490858078003</p>
<p>Below is the CPU utilization graph. Looks like one process occupies half of the cores, and 2 process uses all of them. But still 2 processes execution time is worse than sequential.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/4/64f14a73e39548cbf99fa50ac66498b998a77154.png" data-download-href="https://discuss.pytorch.org/uploads/default/64f14a73e39548cbf99fa50ac66498b998a77154" title="Screen Shot 2020-02-25 at 11.58.09 AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/4/64f14a73e39548cbf99fa50ac66498b998a77154_2_242x500.png" alt="Screen Shot 2020-02-25 at 11.58.09 AM" data-base62-sha1="eoYI2gdetIAWfrkAoGbNwzPybVq" width="242" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/4/64f14a73e39548cbf99fa50ac66498b998a77154_2_242x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/4/64f14a73e39548cbf99fa50ac66498b998a77154_2_363x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/4/64f14a73e39548cbf99fa50ac66498b998a77154_2_484x1000.png 2x" data-dominant-color="2C3F1F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2020-02-25 at 11.58.09 AM</span><span class="informations">706×1458 69.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

albanD: <p>Hi,</p>
<p>Where do you set <code>torch.set_num_threads(1)</code> ?<br>
You have to set it at the beginning of the <code>Worker()</code> function for it to have an effect on the newly created process.</p>
<p>Checking with <a class="mention" href="/u/mrshenli">@mrshenli</a>, this seems to give the right behavior after setting this.</p>
<p>In particular, by default, pytorch will use all the available cores to run computations on CPU. So if you launch two processes to do this at once, then they will fight for the CPU and most likely slow each other down.</p>

Petros_Christodoulou: <p>thanks a lot. I was setting the number of threads in the parent process rather than the worker processes. It seems to resolve it if i set them in each of the worker processes</p>