About DistributedDataParallel

lzkzls: <p>When I trained my program by 4 GPUs, like this:<br>
CUDA_VISIBLE_DEVICES=0,1,2,3 python3 -m torch.distributed.launch --nproc_per_node=4 train.py<br>
there would be an error:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/a/1aba93e75a41f52812a188a289955a6a0df28359.png" data-download-href="https://discuss.pytorch.org/uploads/default/1aba93e75a41f52812a188a289955a6a0df28359" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/a/1aba93e75a41f52812a188a289955a6a0df28359_2_690x395.png" alt="image" data-base62-sha1="3Os9CE6Ub4sknOJpowdhYSDlKJP" width="690" height="395" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/a/1aba93e75a41f52812a188a289955a6a0df28359_2_690x395.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/a/1aba93e75a41f52812a188a289955a6a0df28359_2_1035x592.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/a/1aba93e75a41f52812a188a289955a6a0df28359_2_1380x790.png 2x" data-dominant-color="090909"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2469×1415 243 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
When using nn.DataParallel (), the program can be run. How can I locate this problem?<br>
The snippet of my program is:<br>
dist.init_process_group(backend=‘nccl’, init_method=“env://”)<br>
generator_model = VideoGenerator().cuda(args.local_rank)<br>
discriminator_1 = Discriminator_1().cuda(args.local_rank)<br>
discriminator_2 = Discriminator_2().cuda(args.local_rank)<br>
discriminator _3 = Discriminator_3().cuda(args.local_rank)<br>
generator_model = torch.nn.parallel.DistributedDataParallel(generator_model,device_ids=[args.local_rank])<br>
discriminator_1 = torch.nn.parallel.DistributedDataParallel(discriminator_1,device_ids=[args.local_rank])<br>
discriminator_2 = torch.nn.parallel.DistributedDataParallel(discriminator_2,device_ids=[args.local_rank])<br>
discriminator_3 = torch.nn.parallel.DistributedDataParallel(discriminator_3,device_ids=[args.local_rank])<br>
generator_model.train()<br>
train_sampler = …<br>
train_loader = …<br>
for i, batches in enumerate(train_loader):<br>
first_images, cut_audios = batches<br>
first_images = first_images.float().cuda(args.local_rank)<br>
cut_audios = cut_audios.float().cuda(args.local_rank)<br>
gen_video = generator_model(first_images, cut_audios)  #（line 175）<br>
…</p>

maudzung: <p>Hi,<br>
You can apply the example from <a href="https://github.com/pytorch/examples/blob/59caa16a29476f290d2e4d38162a89372e5ccb15/imagenet/main.py#L156" rel="nofollow noopener">here</a> to your code.</p>

lzkzls: <p>Thanks for your reply, I have read the example you offered, but the error still exists in my program.</p>

mrshenli: <p>The error “NoneType object is not callable” seems to suggest that <code>generator_model</code> somehow becomes None? Can you verify that?</p>
<p>Another thing that stands out is the following:</p>
<pre><code class="lang-python">generator_model = torch.nn.parallel.DistributedDataParallel(generator_model,device_ids=[args.local_rank])
discriminator_1 = torch.nn.parallel.DistributedDataParallel(discriminator_1,device_ids=[args.local_rank])
discriminator_2 = torch.nn.parallel.DistributedDataParallel(discriminator_2,device_ids=[args.local_rank])
discriminator_3 = torch.nn.parallel.DistributedDataParallel(discriminator_3,device_ids=[args.local_rank])
</code></pre>
<p>Any reason for creating 4 DDP instances in one process?</p>
<p>To help us investigate, it will be helpful if you could share a self-contained minimal reproduce-able example.</p>
<p>BTW, could you please add a “distributed” tag to questions related to distributed training so that people working on that can get back to you promptly?</p>

lzkzls: <p>Thank you so much! When I created 4 DDP instances in four different processes using argument process_group, this error disappeared. And could you give me some explanation about the affecting of precess_group？  Why we should create different processes for different DDP instances?</p>

mrshenli: <p>Sure, D</p>
<aside class="quote no-group" data-username="lzkzls" data-post="5" data-topic="87260" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/f14d63/48.png" class="avatar"> lzkzls:</div>
<blockquote>
<p>Thank you so much! When I created 4 DDP instances in four different processes using argument process_group, this error disappeared. And could you give me some explanation about the affecting of precess_group？ Why we should create different processes for different DDP instances?</p>
</blockquote>
</aside>
<p>Sure. When creating DDP instances in the same process without specifying <code>process_group</code> argument, those DDP instances will share the same default process group. If you use these DDP instances interleavingly, it’s likely to cause communication de-synchronization, which would lead to hang or crash (see <a>this</a> brief explanation of DDP implementation). So if you would like to create multiple DDP instances on the same process for different models, you can try provide different process group instances.  You can create new groups using the <a href="https://pytorch.org/docs/stable/distributed.html#torch.distributed.new_group" rel="noopener nofollow ugc"><code>new_group</code></a> API.</p>

lzkzls: <p>Thank you sincerely! But I have another problem: <a href="https://discuss.pytorch.org/t/torch-nn-parallel-distributeddataparallel-problem-about-nonetype-error-calledprocesserror-backward/87546" class="inline-onebox">torch.nn.parallel.DistributedDataParallel() problem about "NoneType Error"\ CalledProcessError\backward</a>, could you give me some advice?</p>

mrshenli: <p>Sure, commented in that post.</p>