RAM memory leak during hvd distributed training

kevin_zhang: <p>Hi,</p>
<p>So I am running horovod distributed pytorch training on a machine with 16 GPUs. I see that the RAM constantly increases, and once it reaches 300GB horovod just errors out:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/f/bf39d417cb951f3f09ef10d1485eb6a2904c7434.png" data-download-href="https://discuss.pytorch.org/uploads/default/bf39d417cb951f3f09ef10d1485eb6a2904c7434" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/f/bf39d417cb951f3f09ef10d1485eb6a2904c7434_2_690x455.png" alt="image" data-base62-sha1="rhF5ODX8UA6XB79qRx8TZHZXwi0" width="690" height="455" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/f/bf39d417cb951f3f09ef10d1485eb6a2904c7434_2_690x455.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/f/bf39d417cb951f3f09ef10d1485eb6a2904c7434_2_1035x682.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/f/bf39d417cb951f3f09ef10d1485eb6a2904c7434.png 2x" data-dominant-color="FBFBFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1378×910 41.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I tried to debug this issue with python memory profiler using only 1 GPU. I did a few training iterations, and here is the output from the profiler:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/6/c6a2ffba01f5bbbfc005ff67618a2c224c19e520.png" data-download-href="https://discuss.pytorch.org/uploads/default/c6a2ffba01f5bbbfc005ff67618a2c224c19e520" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/6/c6a2ffba01f5bbbfc005ff67618a2c224c19e520_2_591x499.png" alt="image" data-base62-sha1="sldLi0I774B6kcPnVUZR9Zpbnig" width="591" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/6/c6a2ffba01f5bbbfc005ff67618a2c224c19e520_2_591x499.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/6/c6a2ffba01f5bbbfc005ff67618a2c224c19e520_2_886x748.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/6/c6a2ffba01f5bbbfc005ff67618a2c224c19e520.png 2x" data-dominant-color="F2F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1148×970 164 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>If I interpret this correctly, whenever I do <code>loss = model(...)</code>, 63 MiB memory gets used and never gets returned (check increment column). So the problem seems to be this issue? Furthermore even after I included <code>del loss</code> on the bottom, the 63MiB is still not returned.</p>
<p>Can anyone share their insight on the root cause?</p>
<p>Thanks</p>

sebk26: <p>Hi <a class="mention" href="/u/kevin_zhang">@kevin_zhang</a>, did you fixed your memory leak? Or how did you avoid it?</p>