ReduceLROnPlateau usage with DistributedDataParallel?

username: m-decoster
cooked: <p>I am implementing a model and my model + data does not fit on a single GPU. I am using DistributedDataParallel because the documentation recommends it over DataParallel.</p>
<p>My model appears to work now (it can overfit), but I am unsure about how I should use ReduceLROnPlateau.</p>
<p>Is it safe to simply call <code>scheduler.step(validation_accuracy)</code> regardless of rank? Or should I only call it on rank 0 and broadcast the resulting learning rate to the other processes (and how)?</p>

username: m-decoster
cooked: <p>I’ve currently implemented something like this, does this look correct?</p>
<pre><code>dist.barrier()  # Synchronize, making sure all processes have reached the end of this epoch.
acc_tensor = torch.tensor(val_acc)
dist.all_reduce(acc_tensor, op=ReduceOp.SUM)
scheduler.step(acc_tensor.item() / world_size)
</code></pre>
<p>In my case world size is the number of processes and the number of GPUs, so this averages the accuracy along all processes and uses it to update the scheduler.</p>
<p>I think this is correct, I am unsure if the barrier is needed. I assume so because otherwise some processes may be lagging behind and not have updated <code>val_acc</code> yet.</p>

username: agolynski
cooked: <p>Hello</p>
<p>You’d need to initialize process group for DDP and it will synchronize workers for the first iteration.</p>
<p>Not sure why you are trying to call dist.all_reduce() here, normally DDP will handle model parameters for you.</p>
<p>Your scheduler is similar to optimizer in this context?</p>
<blockquote>
<p>I think this is correct, I am unsure if the barrier is needed. I assume so because otherwise some processes may be lagging behind and not have updated val_acc yet.</p>
</blockquote>
<p>You generally don’t need to synchronize your workers, e.g.<br>
<a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/ddp_tutorial.html</a><br>
(barriers are only used to synchronize while checkpointing)</p>

username: m-decoster
cooked: <p>The scheduler is <a href="https://pytorch.org/docs/stable/optim.html#torch.optim.lr_scheduler.ReduceLROnPlateau" rel="noopener nofollow ugc">ReduceLROnPlateau</a>, it is used to update the learning rate based on a metric (in my case validation accuracy).</p>
<p>Because <code>val_acc</code> is not a model parameter, I would assume it to be different on every process (because every process has its own mini-batch). Therefore, I need to synchronise it so every process changes the learning rate at the same time.</p>
<p>When I find some time, I will try and work out a minimum example.</p>

username: agolynski
cooked: <p>Sorry, I think I misunderstood your question initially.</p>
<p>(thanks <a class="mention" href="/u/rvarm1">@rvarm1</a> for suggestion!)</p>
<p>Do you do validation on rank 0? If so, you can compute val_acc on rank 0 and broadcast it to all ranks at which point you can run scheduler.step independently, so all ranks are consistent. note that broadcast operation will synchronize the ranks for you.</p>
<p>(by <a class="mention" href="/u/wayi">@wayi</a>)<br>
maybe you can also consider using something like <a href="https://arxiv.org/pdf/2007.05105.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/2007.05105.pdf</a><br>
and<br>
<a href="https://github.com/facebookresearch/fairscale#adascale-sgd" class="inline-onebox" rel="noopener nofollow ugc">GitHub - facebookresearch/fairscale: PyTorch extensions for high performance and large scale training.</a></p>