Training slower when using multi GPUs

TANKUN_LI: <p>Hi, I am using pytorch to train a GAN.</p>
<p>When I just used one GPU, training speed is 0.55 sec/batch with batch size = 8.<br>
However, once I used 2 GPUs (nn.DataParallel()), the speed became 1.4 sec/batch with batch size = 16 (batch size 8 in each GPU). Also when I used 4 GPU with batch size = 32 (batch size 8 each GPU), speed was 3.4 sec/batch. I checked the all the GPU utils are larger than 90%.</p>
<p>I read a lot about this and I know that using multi GPU can be slower because of communication between GPUs. But compared to single GPU, multi GPU are way too slower. There must exist some problem may be related to my Network.</p>
<p>In my code, there are 4 networks in total, such as:</p>
<pre><code class="lang-auto">g = Generator()
d = Discriminator()
a = Detector()
b = Segment()

g = nn.DataParallel(g).to('cuda')
d = nn.DataParallel(d).to('cuda')
a = nn.DataParallel(a).to('cuda')
b = nn.DataParallel(b).to('cuda')

img = torch.rand([1, 3, 256, 245]).to('cuda')
temp1 = a(img)
temp2 = b(temp1)
result = g(temp2)
out_fake = d(result)
</code></pre>
<p>Do you think that may be there are too many separated networks that slow down the communication speed between GPUs?</p>

ptrblck: <aside class="quote no-group" data-username="TANKUN_LI" data-post="1" data-topic="84732">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tankun_li/48/11956_2.png" class="avatar"> TANKUN_LI:</div>
<blockquote>
<p>Do you think that may be there are too many separated networks that slow down the communication speed between GPUs?</p>
</blockquote>
</aside>
<p>This might be the case, as you would have to scatter and gather the results for each model.<br>
A quick check would be to wrap all models in an <code>nn.Sequential</code> container and pass this to <code>nn.DataParallel</code>.</p>
<p>We generally recommend to use DistributedDataParallel with a single process per device, as it would give you the best speedup.</p>

Sergius_Liu: <p>I also ran into similar issue that  4 gpu with  4 x batch_size   almost cost 4 times processing time of  1 gpu with 1x batch_size iteration.</p>
<p>For this statement:</p>
<aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="84732">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>use DistributedDataParallel with a single process per device</p>
</blockquote>
</aside>
<p>Could you give an examples?<br>
I am not very sure what "single process per device " mean…</p>

mrshenli: <p>Hey <a class="mention" href="/u/sergius_liu">@Sergius_Liu</a></p>
<p>Here is an example: <a href="https://pytorch.org/docs/stable/notes/ddp.html" class="inline-onebox" rel="noopener nofollow ugc">Distributed Data Parallel — PyTorch 2.1 documentation</a></p>
<blockquote>
<p>I am not very sure what "single process per device " mean…</p>
</blockquote>
<p>There are two modes in DDP: Single-Process Multi-Device mode (this is retiring) and Single-Process Single-Device mode (this is recommended). Please see <a href="https://github.com/pytorch/pytorch/issues/47012" rel="noopener nofollow ugc">this issue</a> for more context, and let us know if you have concerns. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

Sergius_Liu: <p>Thanks.<br>
Actually, I found the dataloader could be the potential bottle neck.<br>
The loading time is nearly proportional to the batchsize.<br>
I will make another thread about this to focus on the loading time issue.</p>

TANKUN_LI: <p>Sorry for the late response. I change it to <code>nn.Sequential</code> and it is faster.</p>
<p>Thx!</p>

pritamqu: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/sergius_liu">@Sergius_Liu</a><br>
could you please explain how did you put 4 different models in a <code>nn.Sequential</code> container, and then calling different submodules in their forward pass. I was trying to do that, but not sure about <code>forward</code> part.</p>
<pre><code class="lang-auto">class GAN(nn.Module):

    def __init__(self, args):
        super().__init__()
        
        g1 = Generator()
        g2 = Generator()
        d1 = Discriminator()
        d2 = Discriminator()
        
        self.model = nn.Sequential(g1, g2, d1, d2)
        self.model.cuda(args.device)
        self.model = torch.nn.parallel.DistributedDataParallel(self.model, [self.args.gpu])
        init_weights(self.model)

    def forward(self, x):
        ## not sure how can I access individual model and perform forward pass here
        
        return

</code></pre>
<p>could you please share a minimal example? also, can we achieve the same thing with <code>nn.ModuleDict</code>? please see the below code, i was trying something like that:</p>
<pre><code class="lang-auto">class Network(nn.Module):

    def __init__(self, args):
        super().__init__()        
        
        g1 = Generator()
        g2 = Generator()
        d1 = Discriminator()
        d2 = Discriminator()
        
        self.model = nn.ModuleDict()
        
        self.model.Gab = g1 
        self.model.Gba = g2
        self.model.Da = d1
        self.model.Db = d2
        
        self.args = args
        self.model.cuda(args.device)
        
        def parallelize(self):
            
            if self.args.distributed:
                self.model = torch.nn.parallel.DistributedDataParallel(self.model, [self.args.gpu])
                init_weights(self.model)

                # for model_key in self.model:
                #     if self.model[model_key] is not None:
                #         self.model[model_key] = torch.nn.parallel.DistributedDataParallel(self.model, [self.args.gpu])
                #         init_weights(self.model[model_key])
                        
            else:
                init_weights(self.model)
                # for model_key in self.model:
                #     if self.model[model_key] is not None:
                #         init_weights(self.model[model_key])
                
        def forward(self, x, ) :
                **# agiain I am not sure about this part..** 

</code></pre>