Distributed Parallel, one machine multi gpu multi process?

username: Shiro
cooked: <p>Hi everyone,</p>
<p>I am trying to train a model with one machine, but with multi gpus. Until now, I was using the nn.DataParallel which works well, but it seems a bit slow to me so I would like to use the DistributedDataParallel instead.</p>
<p>However, I am not sure to understand clearly how to use this function (I have some weird results, the training takes 10x much more time than DataParallel).<br>
In fact, I am not sure which gpus have to load the model/batch and compute the loss function ?<br>
Moreover with the code below, my training is slower and I saw on nvidia-smi  a weird behavior. Instead of having ONE process on each gpu, I have two process for each gpu (I have two gpus, but I have 4 process) .</p>
<p>My second issue is if I increase the number of workers in the dataloader, I have a dataloader pid killed error .<br>
Am I doing something wrong ?</p>
<pre><code class="lang-auto">import torch
import torch.distributed as dist
import torch.nn as nn
import torch.optim as optim
import torch.multiprocessing as mp
from apex.parallel import DistributedDataParallel as DDP_apex
from torch.nn.parallel import DistributedDataParallel as DDP

def run(gpu, args):
	rank = gpu                   
        dist.init_process_group(                                   
    	        backend='nccl',                                         
   		init_method='tcp://localhost:1088', #'env://',                                   
    	        world_size=args.world_size,                              
    	       rank=rank                                               
              )      
	trainset = ...
	testset = ...
	
	################################################################
        train_sampler = torch.utils.data.distributed.DistributedSampler(
    	trainset,
    	num_replicas=args.world_size,
    	rank=rank
    )

    ################################################################
        test_sampler = torch.utils.data.distributed.DistributedSampler(
    	testset,
    	num_replicas=args.world_size,
    	rank=rank
    )

	trainloader = torch.utils.data.DataLoader(trainset, batch_size=args.batch_size, shuffle=True,                        num_workers=args.workers, pin_memory=False, drop_last= True )
        testloader = torch.utils.data.DataLoader(testset, batch_size=args.batch_size, shuffle=False, num_workers=args.workers, pin_memory=False)
	optim_params = list(filter(lambda p: p.requires_grad, net.parameters()))
	optimizer = optim.Adam(optim_params, lr=args.lr, betas=(0.9, 0.999), eps=1e-08, weight_decay=args.weight_decay, amsgrad=True)
	net = 2D_CNN()
	net = net.to(args.gpus[0])
	
	net = DDP(net, device_ids=args.gpus)
	
	train(net, optimizer,  trainloader, testloader, args, gpu) # function which iterate accross the dataloader and do the forward/backward/step

if __name__ =="__main__":
    args.nodes = 1 # one single machine
    args.gpus = [0,1,2]
    #########################################################
    args.world_size = len(args.gpus) * args.nodes                #
    os.environ['MASTER_ADDR'] = 'localhost'              #
    os.environ['MASTER_PORT'] = '8888'                      #
    print(args.gpus)
    print(os.environ['MASTER_PORT'] )
    mp.spawn(run, nprocs=len(args.gpus), args=(args,))         #
</code></pre>

username: kirk86
cooked: <p>Related to my question, read <a href="https://discuss.pytorch.org/t/distributed-data-parallel-slower-than-data-parallel/72052/4">here</a></p>

username: mrshenli
cooked: <p>One requirement for DistributedDataParallel is that, you need to set <a href="https://github.com/pytorch/pytorch/blob/v1.4.0/torch/nn/parallel/distributed.py#L171" rel="nofollow noopener"><code>device_ids</code></a> properly or use <code>CUDA_VISIBLE_DEVICES</code> env var to configure them properly to make sure that one process only works on one GPU. Otherwise, by default, each process will try to use all visible GPUs.</p>

username: Shiro
cooked: <p>Thanks for your replies.</p>
<p><a class="mention" href="/u/mrshenli">@mrshenli</a>  How can I configure the devices_ids if I want to have one process = one gpu ?<br>
In my case, I have tried to change device_ids by the gpu number (which is equal to the rank value in my case) such as<code> device_ids = [gpu]</code> and call transfer the model in the first gpu <code>model.to(args.gpus[0])</code> .<br>
But unfortunately, I got that error : <code>RuntimeError: Expected tensor for argument #1 'input' to have the same device as tensor for argument #2 'weight'; but device 1 does not equal 0 (while checking arguments for cudnn_convolution)</code></p>
<p>P.S : it seems to “work” now, I forgot to cast the loss function on the good gpu. But, when I increase the number of workers for the dataloader (&gt;0) it creates the error : <code>RuntimeError: DataLoader worker (pid(s) 16451) exited unexpectedly</code> . (and the training took above 24 hours). I will try with the apex distributed and then again compare with DataParallel.</p>

username: kirk86
cooked: <p>Please report back if you find that distributed is faster than data parallel (i.e. time it and lets us know). My intuition says that the claim is false but I haven’t seen anywhere on the docs a proper MWE on how to properly use distributed for each individual case in combination with <code>launch.py</code>.</p>

username: mrshenli
cooked: <p>cc <a class="mention" href="/u/vincentqb">@vincentqb</a> on RuntimeError from DataLoader</p>

username: mrshenli
cooked: <p><a href="https://pytorch.org/docs/master/notes/ddp.html#example" rel="nofollow noopener">This</a> is a minimum DDP example. Posting it here in case it is useful for future readers.</p>

username: kirk86
cooked: <p>That doesn’t cover <code>launch.py</code> examples.</p>
<p>I would like to see examples using <code>launch.py</code> covering the following two separate use cases<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/4/042c3675a7e444492d1916d38c3e1fee479ff405.png" data-download-href="https://discuss.pytorch.org/uploads/default/042c3675a7e444492d1916d38c3e1fee479ff405" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/4/042c3675a7e444492d1916d38c3e1fee479ff405.png" alt="image" data-base62-sha1="AUDnNSRjfyFUFGj0Uk6LgXLkUJ" width="479" height="499" data-dominant-color="F6F8F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">583×608 24.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: Shiro
cooked: <p><a class="mention" href="/u/mrshenli">@mrshenli</a> Thanks for the link.</p>
<p>I see that my training take exactly the same amount of time with 1 ou 2 gpus, so I was wondering, when we use Distributed process, do we have to divide the original number of iteration by the number of gpus used ?</p>

username: mrshenli
cooked: <p>Hey <a class="mention" href="/u/kirk86">@kirk86</a>, could you please create an issue on GitHub for us to track this request? Thanks!</p>

username: mrshenli
cooked: <p><a class="mention" href="/u/shiro">@Shiro</a></p>
<p>If each process runs the same amount of iterations with each iteration consuming the same amount of data, using 2GPUs might actually take longer, because there will be additional communication overhead between the two GPUs. But in this case, your model is actually trained using 2X number of batches.</p>
<p>Reducing the number of iterations should work, or you can also reduce the batch size. One thing to note is that, this might also call for additional tuning on the learning rate or other configs. Some relevant discussions are available <a href="https://discuss.pytorch.org/t/should-we-split-batch-size-according-to-ngpu-per-node-when-distributeddataparallel/72769/8">here</a>.</p>

username: kirk86
cooked: <p><a href="https://github.com/pytorch/pytorch/issues/35160" rel="nofollow noopener">Done</a>!</p>