Multiprocessing slows dataloaders at the each beginning epoch

username: ralzq01
cooked: <p>Hi, I’m currently using <code>torch.multiprocessing</code> for sending the outputs of a neural network to another process. For simple discussion, I have two processes: the first one is for loading training data, forwarding network and sending the results to the other one, while the other one is for recving the results from the previous process and handling the results.</p>
<p>The question I get troubled in is that, multiprocesses slow down the data loader. At every begining of each epoch, the forwarding process will unexpectedly get stucked for a while and then continue forwarding. When I cancel the multiprocess (i.e. just forward network and don’t send data across processes), all things go smoothly, and the stucked phenomenon at every begining epoch disappears.</p>
<p>Below is a python3 simple script (runs on one GTX 1080) which loads a pretrained network and use the dataset of Cifar10 with arguments of  <code>num_workers=2</code> and <code>pin_memory=True</code>. You will observe the program gets stucked between <code>for epoch in range(total_epoch)</code> and <code>for step, datas in enumerate(trainloader)</code></p>
<p>Anyone has any idea on why this happens and how to solve this? (I am doubting it is caused by limits of shared memory, but still don’t know how to avoid this)</p>
<pre><code class="lang-auto">import torch.multiprocessing as mp

def main():

  os.environ['CUDA_VISIBLE_DEVICES'] = '0'
  device = torch.device('cuda:0')

  model_name = 'vgg16'
  dataset = 'cifar10'

  total_epoch = 20
  batch_size = 128

  # get model and datasets
  model = get_a_pretrained_model
  trainloader = get_cifar10_trainloader # numworkers=2, pin_memory=True
  for param in model.parameters():
    param.requires_grad = False
  model = model.to(device)
  model.eval()

  mp.set_start_method('spawn')
  buffer = mp.Queue()
  recv_proc = mp.Process(target=recv, args=(buffer,))
  recv_proc.start()

  # forwarding model and send results
  for epoch in range(total_epoch):
    # program will get stucked here
    for step, datas in enumerate(trainloader):
      inputs, labels = datas
      inputs, labels = inputs.to(device), labels.to(device)
      outs = model(inputs)
      buffer.put(outs)
      if step % 20 == 0:
        print('Epoch [{}]\t Step [{}]'.format(epoch, step))
  buffer.put(None)

  recv_proc.join()
  

def recv(buffer):
  while True:
    recved = buffer.get()
    if recved is None:
      break
    # do something with recved
    del recved


if __name__ == '__main__':
  main()
</code></pre>

username: auroua
cooked: <p>I encountered the same problem. Have you solved this problem?</p>

username: chen_dongdong
cooked: <p>Same problem encountered, can anybody answer it?</p>

username: sailist
cooked: <p>update pytorch to 1.7, then the arg <code>persistent_workers</code> will be helped. See this link <a href="https://discuss.pytorch.org/t/what-are-the-dis-advantages-of-persistent-workers/102110" class="inline-onebox">What are the (dis) advantages of persistent_workers</a></p>

username: AndreyBocharnikov
cooked: <p>Thank you so much man, been facing this problem for a month, so glad to find you answer!</p>

username: aowei
cooked: <p>I changed the start method to fork <strong>mp.set_start_method(‘fork’)</strong>. It solved the problem.</p>