Gradient_as_bucket_view does not seem to do anything

kleingeo: <p>I am trying a large model using DDP with a single batch on 4 GPUs on a single node. I am trying to reduce GPU memory usage as much as possible and have been trying to set <code>gradient_as_bucket_view=True</code> to reduce CUDA memory overhead. However, when setting <code>gradient_as_bucket_view</code> to either <code>True</code> or <code>False</code>, I am seeing about 2GB of additional overhead per card.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/0/a0f2e7eea13453074c8d9f9d3cb086b12dc2bd96.png" data-download-href="https://discuss.pytorch.org/uploads/default/a0f2e7eea13453074c8d9f9d3cb086b12dc2bd96" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/0/a0f2e7eea13453074c8d9f9d3cb086b12dc2bd96.png" alt="image" data-base62-sha1="mXOOM1o9GQDMjlVGk11T8FzCYCi" width="546" height="500" data-dominant-color="2D2D2D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">566×518 11 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I am also not sure if this is related, but in PyCharm, the argument for <code>gradient_as_bucket_view</code> is indicated as an <em>unexpected argument</em>, but I am not sure why this is so.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/e/4ea9f4b68dc3e3bd84458935c3bedff211df0403.png" alt="image" data-base62-sha1="bdTn2fJfuKrBelS3AdaNhMnylPR" width="584" height="62"></p>
<p>I am running torch==1.10.1+cu11.3 installed through pip (cannot use conda as it is through a HPC).</p>

pritamdamania87: <aside class="quote no-group" data-username="kleingeo" data-post="1" data-topic="140906">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kleingeo/48/21537_2.png" class="avatar"> kleingeo:</div>
<blockquote>
<p>However, when setting <code>gradient_as_bucket_view</code> to either <code>True</code> or <code>False</code>, I am seeing about 2GB of additional overhead per card.</p>
</blockquote>
</aside>
<p>What is the 2GB additional overhead in comparison to? Are you comparing to when you don’t use DDP at all?</p>
<p>cc <a class="mention" href="/u/yanli_zhao">@Yanli_Zhao</a></p>

kleingeo: <aside class="quote no-group" data-username="pritamdamania87" data-post="2" data-topic="140906">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pritamdamania87/48/28891_2.png" class="avatar"> pritamdamania87:</div>
<blockquote>
<p>hat is the 2GB additional overhead in comparison</p>
</blockquote>
</aside>
<p>Its a lot higher when compared to another HPC server I use (which usually is at 980 MB overhead), but it is using a different CUDA version, and more importantly an unoptimized version of PyTorch which is the likely culprit.</p>
<p>Should it not be reduced with <code>gradient_as_bucket_view</code>? Or am I just confused with the parameter?</p>

pritamdamania87: <aside class="quote no-group" data-username="kleingeo" data-post="3" data-topic="140906">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kleingeo/48/21537_2.png" class="avatar"> kleingeo:</div>
<blockquote>
<p>Should it not be reduced with <code>gradient_as_bucket_view</code>? Or am I just confused with the parameter?</p>
</blockquote>
</aside>
<p>The memory should be reduced when you specify <code>gradient_as_bucket_view</code>. After initializing DDP, can you run a few iterations and then look at the memory consumption? Also, typically looking at nvidia-smi is not a good indication of memory usage due to the caching allocator (see <a href="https://pytorch.org/docs/stable/notes/cuda.html#cuda-memory-management" class="inline-onebox" rel="noopener nofollow ugc">CUDA semantics — PyTorch 2.1 documentation</a>). I’d suggest using <a href="https://pytorch.org/docs/stable/generated/torch.cuda.memory_allocated.html#torch.cuda.memory_allocated" class="inline-onebox" rel="noopener nofollow ugc">torch.cuda.memory_allocated — PyTorch 2.1 documentation</a> or <a href="https://pytorch.org/docs/stable/generated/torch.cuda.max_memory_allocated.html#torch.cuda.max_memory_allocated" class="inline-onebox" rel="noopener nofollow ugc">torch.cuda.max_memory_allocated — PyTorch 2.1 documentation</a> to track this.</p>

kleingeo: <p>Ok, I’ll try that then with 2 GPUS. Is there a way to reduce the overhead from DDP?</p>

pritamdamania87: <p>Setting <code>gradient_as_bucket_view=True</code> is the primary way to avoid some of the overhead. Also note that you need to run a few training iterations for the memory savings to be applied.</p>