Logging loss value in DDP training

username: amirhf
cooked: <p>Hello,</p>
<p>I am reviewing the <a href="https://github.com/pytorch/examples/blob/master/imagenet/main.py" rel="noopener nofollow ugc">pytorch imagenet</a> example in the repos and I have trouble comprehending the loss value that is returned by the <code>criterion</code> module. In <a href="https://github.com/pytorch/examples/blob/0f0c9131ca5c79d1332dce1f4c06fe942fbdc665/imagenet/main.py#L291" rel="noopener nofollow ugc">Line 291</a>, is the loss that is recorded later for only one process? Is summing and averaging all losses across all processes using <code>ReduceOp.SUM</code> a better alternative? For example, when I want to save my model or simply log the metric, I would like to do it based on the average loss values across all processes.</p>
<p>In other words, is <code>losses.update(loss.item(), images.size(0))</code> only saving the loss value for one process?</p>

username: mrshenli
cooked: <aside class="quote no-group" data-username="amirhf" data-post="1" data-topic="100429">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/amirhf/48/5634_2.png" class="avatar"> amirhf:</div>
<blockquote>
<p>is the loss that is recorded later for only one process?</p>
</blockquote>
</aside>
<p>Hey <a class="mention" href="/u/amirhf">@amirhf</a>, if you are using <code>DistributedDataParallel</code>, yep, this is the local loss within one process. And different process can have different loss values.</p>
<blockquote>
<p>Is summing and averaging all losses across all processes using <code>ReduceOp.SUM</code>  a better alternative?</p>
</blockquote>
<p>This will give you the global loss, but will also introduce one more communication per iteration. If this is just for logging purpose, will it be sufficient if the logging is done every <code>n</code> iterations, so that there will be a smaller amortized comm overhead?</p>

username: amirhf
cooked: <p>Hi <a class="mention" href="/u/mrshenli">@mrshenli</a>,</p>
<p>Thank you for your response! So the concern is that the reduce operation is an overhead. Yes, so I would just log the global loss every few iterations. Something along the lines of <code>if iteration % 200 == 0: reduce and log</code>. Is that going to be okay?</p>

username: seungjun
cooked: <p>Hi <a class="mention" href="/u/amirhf">@amirhf</a>,</p>
<p>I synchronize the loss record at every epoch, which is about 100 ~ 1500 iterations varying by the dataset I use. I didn’t see much performance degradation with that. The gains in DistributedDataParallel was bigger (compared to DataParallel) than the loss in the GPU communication time in my experience.</p>
<p>You can refer to my code <a href="https://github.com/SeungjunNah/DeepDeblur-PyTorch/blob/master/src/loss/__init__.py#L257" rel="noopener nofollow ugc">here</a>.</p>

username: mrshenli
cooked: <p>Yep, that should be infrequent enough.</p>
<p>There are also ways to hide this communication delay by setting <code>async_op=True</code> when launching the <code>dist.reduce</code>, and only wait on the returned handle after the next forward pass. This would allow the communication of <code>dist.reduce</code> to overlap with the next fw pass.</p>