Longet training time, lower batch size, and OOM error with distributed data parallel

meshghi: <p>Hello! I’m trying to use DDP instead of DP for my model training. However, DDP forces me to either use a smaller batch size or longer training time. I have a few experiments as follows, and I appreciate some insights:</p>
<p><strong>Longer training time:</strong></p>
<ul>
<li>Triaining on one node with 8 GPUs: when I use DP and batch size 40, my understanding is that pytorch uses it to send batches of size 5 to each of 8 GPUs. When I use DDP, I specify batch size 5 which is used for each GPU which again sums up to 40. However, the training time with DDP is longer than DP.</li>
</ul>
<p><strong>Smaller Batch Size:</strong></p>
<ul>
<li>Triaining on two/three nodes with 8 GPUs each, totally 16/24 GPUs: In this setting, I get CUDA Out of Memory even with batch size 1. The model is UNET-VGG16 and each GPU has 16G memory. My assumption is that this is because of the AllReduce action. And that is why going from 8 GPUs in the previous question above to a multi-node setting causes OOM error because it needs to bring 16/24 model parameters to each GPU for synchronization. I even tried <code>gradients_as_bucket_view=True</code>, but it didn’t fix the issue. My question is that if my reasoning is true, the DDP is not helpful since multi node training is not possible even with the small UNET model. But, if my reasoning is wrong, I would appreciate some help to fix my issue with DDP.</li>
</ul>

ptrblck: <aside class="quote no-group" data-username="meshghi" data-post="1" data-topic="155471">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/meshghi/48/33305_2.png" class="avatar"> meshghi:</div>
<blockquote>
<p>Triaining on one node with 8 GPUs: when I use DP and batch size 40, my understanding is that pytorch uses it to send batches of size 5 to each of 8 GPUs. When I use DDP, I specify batch size 5 which is used for each GPU which again sums up to 40. However, the training time with DDP is longer than DP.</p>
</blockquote>
</aside>
<p>Yes, <code>DataParallel</code> will split the batch in <code>dim0</code> and send the chunks to each device. It will introduce additional overheads via the model copies and should be slower than <code>DDP</code>. Could you share a minimal, executable code snippet showing that <code>DDP</code> is slower than <code>DP</code>, please?</p>
<aside class="quote no-group" data-username="meshghi" data-post="1" data-topic="155471">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/meshghi/48/33305_2.png" class="avatar"> meshghi:</div>
<blockquote>
<p>Triaining on two/three nodes with 8 GPUs each, totally 16/24 GPUs: In this setting, I get CUDA Out of Memory even with batch size 1.</p>
</blockquote>
</aside>
<p>I guess you are accidentally creating CUDA contexts on the default device and it’s running out of memory. Try to use <code>torch.cuda.set_device</code> or mask the devices via <code>CUDA_VISIBLE_DEVICES</code>.</p>

meshghi: <p>Thanks for your reply.</p>
<blockquote>
<p>Yes, <code>DataParallel</code> will split the batch in <code>dim0</code> and send the chunks to each device. It will introduce additional overheads via the model copies and should be slower than <code>DDP</code>. Could you share a minimal, executable code snippet showing that <code>DDP</code> is slower than <code>DP</code>, please?</p>
</blockquote>
<p>I have the following code that runs slow on my data. I haven’t tested this on small sample data. Sorry that my code is not executable.</p>
<pre><code class="lang-auto">def main(gpu, cfg):
    """
    The gpu from the main function's argument goes from 0 to world_size-1. Therefore, for the local
    rank,  for one node I used cfg.gpu = gpu, but for multi node I use cfg.gpu = gpu // len(cfg.gpus)
    to create gpu local number from 0 to 7 per node as local rank.
    """
    cfg.gpu = gpu # local rank in the case of one node training
    cfg.gpu = gpu // len(cfg.gpus) # local rank in the case of multi node training
    cfg.rank = gpu # global rank

    torch.distributed.init_process_group(backend='nccl',
                                         init_method=cfg.dist_url,
                                         world_size=cfg.worl_size,
                                         rank=cfg.rank)

    torch.cuda.set_device(cfg.gpu)
    torch.distributed.barrier()

    train_dataset = getDataset('train')
    valid_dataset = getDataset('valid')

    train_sampler = torch.utils.data.distributed.DistributedSampler(train_dataset,
                                                                    num_replicas=cfg.world_size,
                                                                    rank=cfg.rank,
                                                                    shuffle=False,
                                                                    drop_last=True)

    train_dataloader = torch.utils.data.DataLoader(train_dataset,
                                                   collate_fn=my_collate,
                                                   worker_init_fn=worker_init_fn,
                                                   sampler=train_sampler)

    valid_dataloader = torch.utils.data.DataLoader(valid_dataset,
                                                   collate_fn=my_collate,
                                                   sampler=None)

    model = Net()
    model.cuda(cfg.gpu)

    model = torch.nn.parallel.DistributedDataParallel(model,
                                                      device_ids=[cfg.gpu],
                                                      output_device=cfg.gpu,
                                                      gradient_as_bucket_view=True)

    optimizer = optimizer()
    criterion = criterion().cuda(cfg.gpu)
    scheduler = scheduler()

    device = torch.device("cuda:{}".format(cfg.gpu))
    runner = Runner(model, 
                    model_device=device, 
                    optimizer, 
                    criterion, 
                    scheduler, 
                    train_dataloader, 
                    valid_dataloader)

    runner.train()

    torch.distributed.destroy_process_group()
 
if __name__== "__main__":
    cfg = getConfig()
    os.environ["MASTER_ADDR"]=hostname
    os.environ["MASTER_PORT]=portNumber
    os.environ["CUDA_VISIBLE_DEVICES"]=",".join(map(str,cfg.gpus)) #cfg.gpus=[0,1,2,3,4,5,6,7] per node
    cfg.["dist_url"]=f'env://{hostname}:{portNumber}'
    torch.multiprocessing.spawn(main, nprocs=cfg.world_size, args=(cfg,))
</code></pre>
<blockquote>
<p>I guess you are accidentally creating CUDA contexts on the default device and it’s running out of memory. Try to use <code>torch.cuda.set_device</code> or mask the devices via <code>CUDA_VISIBLE_DEVICES</code>.</p>
</blockquote>
<p>I am using both of these (shown in the code above). I don’t know what else I should do. I’m stuck.</p>

meshghi: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Any suggestion would be great. Thank you!</p>

ptrblck: <p>I can’t really debug the code since it’s not executable, but also note that:</p>
<pre><code class="lang-python">cfg.gpu = gpu // len(cfg.gpus)
</code></pre>
<p>will use <code>cuda:0</code> regardless which value <code>gpu</code> has (assuming it’s in the range of all available GPUs).<br>
so <code>torch.cuda.set_device(cfg.gpu)</code> would use the default device and would fit my suspicion from my previous post.</p>