Weird loss curves using DistributedDataParallel

username: kaizhao
cooked: <p>I’m converting a <code>DataParallel</code> (DP) model into <code>DistributedDataParallel</code> (DDP).</p>
<p>However, the DDP degradates the testing performance and leads to zigzag loss curves during training.</p>
<p>Below is the loss curve during training (blue is DDP and orange is DP):<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/b/4ba011b893a2daf7a32f013d72fb6fbfe8d950e4.png" data-download-href="https://discuss.pytorch.org/uploads/default/4ba011b893a2daf7a32f013d72fb6fbfe8d950e4" title="企业微信截图_16305677882234"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/b/4ba011b893a2daf7a32f013d72fb6fbfe8d950e4_2_690x134.png" alt="企业微信截图_16305677882234" data-base62-sha1="aN0KVX3Bu05hDhRtvzgJPVkfUBC" width="690" height="134" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/b/4ba011b893a2daf7a32f013d72fb6fbfe8d950e4_2_690x134.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/b/4ba011b893a2daf7a32f013d72fb6fbfe8d950e4_2_1035x201.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/b/4ba011b893a2daf7a32f013d72fb6fbfe8d950e4_2_1380x268.png 2x" data-dominant-color="FCFCFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">企业微信截图_16305677882234</span><span class="informations">2131×414 24.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>In general these two behave similar. But when zooming in, the DDP presents periodic zigzags:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/0/d059345a07a999a1e27089824ad93f9ca3600e4a.png" data-download-href="https://discuss.pytorch.org/uploads/default/d059345a07a999a1e27089824ad93f9ca3600e4a" title="企业微信截图_16305679645366"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/0/d059345a07a999a1e27089824ad93f9ca3600e4a_2_690x129.png" alt="企业微信截图_16305679645366" data-base62-sha1="tJ8qSNwXyKxxhLd9UfMZtqX3Sum" width="690" height="129" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/0/d059345a07a999a1e27089824ad93f9ca3600e4a_2_690x129.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/0/d059345a07a999a1e27089824ad93f9ca3600e4a_2_1035x193.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/0/d059345a07a999a1e27089824ad93f9ca3600e4a_2_1380x258.png 2x" data-dominant-color="FBFCFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">企业微信截图_16305679645366</span><span class="informations">2150×402 33.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>My training configurations are:</p>
<ul>
<li>
<p>DP:</p>
<ul>
<li>batch size 48</li>
<li>3 GPUs</li>
</ul>
</li>
<li>
<p>DDP</p>
<ul>
<li>batchsize per GPU: 16</li>
<li>3 GPUs</li>
</ul>
</li>
</ul>
<p>Random seeds are fixed and learning rate and all other configurations are the same.</p>

username: Yanli_Zhao
cooked: <p>DP’s loss is aggregated for batch size 48, DDP’s loss is calculated independently for each batch size 16, so they are expected to be different if you are looking at losses at rank 0</p>