Weird learning stagnation when using DataParallel

kirk86: <p>I’m currently facing a weird behaviour which I cannot explain. I’m training a vgg16 on svhn and training on 1 gpu with SGD and fixed hypeparams I get the following nice results:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/c/c99edc9763b359be5f3f11a5c43ec1899ffb1261.png" alt="image" data-base62-sha1="sLCkwWtU5VRzkZb7CUbKSDBkFPj" width="560" height="458"></p>
<p>Now trying to train the same model with same optimizer and hyperparams as before but using <code>DataParallel</code> it exhibits the following behaviour where the learning process actually stagnates and it doesn’t learn anything.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/2/2df24471697b8c16ff439d2cb49c7e4919d963f2.png" alt="image" data-base62-sha1="6ysxzR0p8J84KfHp0DmveNgRVMS" width="559" height="474"></p>
<p>Even more weird is the fact that if I swap vgg16 for resnet50 it starts learning again.</p>
<p>Anyone has any insights on this, or what it might be going on?</p>
<p>I would expect that if a model <code>M</code> trained on one device with fixed optimizer and hyperams exhibiting good learning behaviour to have the same behaviour when trained with <code>DataParallel</code> using same optimizer and hyperparams.</p>

dsuess: <p>Could you post a minimal working example to reproduce these results? It could be, e.g. that you don’t pass the DataParallel’s parameters to the <code>params</code> argument of the optimizer</p>

kirk86: <p><a class="mention" href="/u/dsuess">@dsuess</a> Thanks for the response!<br>
I’ve actually followed the example <a href="https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html?highlight=data%20parallel" rel="noopener nofollow ugc">here</a> for <code>DataParallel</code>.</p>
<p>Here’s a MWE (trying to avoid putting lot’s of code here)</p>
<pre><code class="lang-auto">model = VGG16
 dataset = CIFAR10

def main():
    model = create_model()
    train_loader = torch.utils.data.DataLoader(...CIFAR10...)
    optimizer = torch.optim.SGD(model.parameters(), lr=0.05, weight_decay=5e-4, momentum=0.9)
     if torch.cuda.device_count() &gt; 1:
         model = torch.nn.parallel.DataParallel(model).to(device)
     else:
         model.to(device)
     train()
     test()
</code></pre>
<aside class="quote no-group" data-username="dsuess" data-post="2" data-topic="55547">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dsuess/48/9946_2.png" class="avatar"> dsuess:</div>
<blockquote>
<p>It could be, e.g. that you don’t pass the DataParallel’s parameters to the <code>params</code> argument of the optimizer</p>
</blockquote>
</aside>
<p>I think that might be the case since in the MWE I’m creating the optimizer before putting the model on DataParallel?</p>
<p>But, it doesn’t explain why the same code works when everything else is constant in the MWE and just swap VGG16 for ResNet50?</p>
<p>Let me give it a try and rearrange the optimizer order after model is sent to DataParallel.</p>
<p>Last question, do you by any chance have any insight on <a href="https://discuss.pytorch.org/t/hanging-distributed-data-parallel-on-interactive-example/55439">this problem</a>?</p>
<p>Thanks!</p>

nooblyh: <p>It looks like a dimension problem to me. Have you check that the batch sizes are at the same position?</p>

kirk86: <aside class="quote no-group" data-username="nooblyh" data-post="4" data-topic="55547">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/eb9ed0/48.png" class="avatar"> nooblyh:</div>
<blockquote>
<p>Have you check that the batch sizes are at the same position?</p>
</blockquote>
</aside>
<p>What do you mean by at the same position?<br>
My understanding is that <code>DataParallel</code> takes data of batch size <code>m</code> and splits them as <code>int(m/num_of_devices)</code> sending to each device it’s own split and a copy of the model?</p>

dsuess: <p>OK, just double checked and your order seems to be correct. I think it’s best if you post a full working example, otherwise we’re just guessing</p>

nooblyh: <p>Sorry, I mean have you rule out the possibility of shape mismatch?</p>

kirk86: <p>Ok so I did check and reordered the code as below:</p>
<pre><code class="lang-auto">model = VGG16
dataset = CIFAR10

def main():
    model = create_model()
    train_loader = torch.utils.data.DataLoader(...CIFAR10...)
     if torch.cuda.device_count() &gt; 1:
         model = torch.nn.parallel.DataParallel(model)
     optimizer = torch.optim.SGD(model.parameters(), lr=0.05, weight_decay=5e-4, momentum=0.9)
     model.to(device)
     train()
     test()
</code></pre>