I increase the batch size but the Memory-Usage of GPU decrease

Teaman: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/7/4739fbfd00766abc2b26a37cb674d025ab6bc96a.png" data-download-href="https://discuss.pytorch.org/uploads/default/4739fbfd00766abc2b26a37cb674d025ab6bc96a" title="1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/7/4739fbfd00766abc2b26a37cb674d025ab6bc96a_2_690x281.png" alt="1" data-base62-sha1="aa688xAtIaIAefpxduEEC8dvkMy" width="690" height="281" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/7/4739fbfd00766abc2b26a37cb674d025ab6bc96a_2_690x281.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/7/4739fbfd00766abc2b26a37cb674d025ab6bc96a_2_1035x421.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/7/4739fbfd00766abc2b26a37cb674d025ab6bc96a_2_1380x562.png 2x" data-dominant-color="B5B6B8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1</span><span class="informations">1815×741 49.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/5/450e63ca95ebe53a7fb569cc79b06da32c5d6c68.png" data-download-href="https://discuss.pytorch.org/uploads/default/450e63ca95ebe53a7fb569cc79b06da32c5d6c68" title="2"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/5/450e63ca95ebe53a7fb569cc79b06da32c5d6c68_2_690x287.png" alt="2" data-base62-sha1="9QTM4Yba3qlBB9fSTSJjh6HGOtW" width="690" height="287" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/5/450e63ca95ebe53a7fb569cc79b06da32c5d6c68_2_690x287.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/5/450e63ca95ebe53a7fb569cc79b06da32c5d6c68_2_1035x430.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/5/450e63ca95ebe53a7fb569cc79b06da32c5d6c68_2_1380x574.png 2x" data-dominant-color="B4B5B7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">2</span><span class="informations">1816×756 53.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/4/c43ac65b84ea52555914e0d900c23b700c2fa628.png" data-download-href="https://discuss.pytorch.org/uploads/default/c43ac65b84ea52555914e0d900c23b700c2fa628" title="3"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/4/c43ac65b84ea52555914e0d900c23b700c2fa628_2_690x291.png" alt="3" data-base62-sha1="rZVvxtBJtvbQNzXnjyxeaHMsl6w" width="690" height="291" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/4/c43ac65b84ea52555914e0d900c23b700c2fa628_2_690x291.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/4/c43ac65b84ea52555914e0d900c23b700c2fa628_2_1035x436.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/4/c43ac65b84ea52555914e0d900c23b700c2fa628_2_1380x582.png 2x" data-dominant-color="B2B4B5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">3</span><span class="informations">1817×768 51.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
As you can see when the batch size is 40 the Memory-Usage of GPU is about 9.0GB, when I increase the batch size to 50, the Memory-Usage of GPU decrease to 7.7GB. And I continued to increase the batch size to 60, and it increase to 9.2GB. Why the Memory-Usage of GPU was so high.According to the common sense, it should be lower than 7.7GB.</p>

ptrblck: <p>The displayed memory usage should show the CUDA context + the actual memory used to store tensors + cached memory + other applications.<br>
Try to check the memory using <code>torch.cuda.memory_allocated()</code> and <code>torch.cuda.memory_cached()</code>.</p>

Teaman: <p>I add the flowing sentence in my code</p>
<pre><code class="lang-auto">                        if (iteration+1)%10 == 0:
                stop = time.time()
                print("epoch: [%d/%d]"%(epoch,EPOCHS), "iteration: [%d/%d]"%(iteration + 1,len(train_dataset)//BATCH_SIZE), "loss:%.4f" % loss.item(),
                      'time:%.4f' % (stop - start))

                print("torch.cuda.memory_allocated: %fGB"%(torch.cuda.memory_allocated()/1024/1024/1024))
                print("torch.cuda.memory_cached: %fGB"%(torch.cuda.memory_cached()/1024/1024/1024))
                start = time.time()
</code></pre>
<p>And the output in the terminal as follow:</p>
<pre><code class="lang-auto">(pt1.2) D:\code\rnn&gt;python train_new.py -m test -b 40
epoch: [0/3000] iteration: [10/92] loss:0.1948 time:28.8928
torch.cuda.memory_allocated: 0.141236GB
torch.cuda.memory_cached: 8.539062GB
epoch: [0/3000] iteration: [20/92] loss:0.0986 time:6.5122
torch.cuda.memory_allocated: 0.141236GB
torch.cuda.memory_cached: 8.539062GB

</code></pre>
<p>(pt1.2) D:\code\rnn&gt;python train_new.py -m test -b 50<br>
epoch: [0/3000] iteration: [10/73] loss:0.1436 time:29.8940<br>
torch.cuda.memory_allocated: 0.144663GB<br>
torch.cuda.memory_cached: 7.197266GB<br>
epoch: [0/3000] iteration: [20/73] loss:0.0644 time:7.6573<br>
torch.cuda.memory_allocated: 0.144663GB<br>
torch.cuda.memory_cached: 7.197266GB</p>
<pre><code class="lang-auto"></code></pre>
<p>(pt1.2) D:\code\rnn&gt;python train_new.py -m test -b 60<br>
epoch: [0/3000] iteration: [10/61] loss:0.1918 time:31.1637<br>
torch.cuda.memory_allocated: 0.151530GB<br>
torch.cuda.memory_cached: 8.666016GB<br>
epoch: [0/3000] iteration: [20/61] loss:0.0936 time:8.8493<br>
torch.cuda.memory_allocated: 0.151408GB<br>
torch.cuda.memory_cached: 8.666016GB</p>
<pre><code class="lang-auto"></code></pre>

ptrblck: <p>Since the allocated memory increases with a higher batch size, it looks fine. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Teaman: <p>yes, but I wonder why the memory cached of b40 will be larger than b50. It will be difficult for me to set the proper batch size.</p>

ptrblck: <p>The cache size might vary e.g. due to cudnn benchmarking and shouldn’t yield an out of memory error, if I’m not mistaken. Are you running out of memory using a smaller batch size?</p>

Youguang_Xing: <p>hello, when batch size increases only by 1(from 4 to 5), why my pyptorch allocated memory still stays the same but the cached memory inscreses?</p>
<div class="md-table">
<table>
<thead>
<tr>
<th style="text-align:center">batch size</th>
<th style="text-align:center">TI allocated</th>
<th style="text-align:center">TI cached</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:center">4</td>
<td style="text-align:center">4674.74 MB</td>
<td style="text-align:center">25890.00 MB</td>
</tr>
<tr>
<td style="text-align:center">5</td>
<td style="text-align:center">4674.74 MB</td>
<td style="text-align:center">31212.00 MB</td>
</tr>
</tbody>
</table>
</div>

ptrblck: <p>This should not be the case since already the input tensor would use more memory, so I guess the increase might be too small and is truncated in your MB view.</p>

Youguang_Xing: <aside class="quote no-group" data-username="ptrblck" data-post="8" data-topic="56684">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>memory</p>
</blockquote>
</aside>
<p>Hi, I’m fine-tuning stable diffusion1-5, and tested these three fine-tune ways, <code>TI</code>, <code>LoRA </code>, <code>LoRA--CTI</code>. Since the model is so big, I do not think the increse is small. I think the computation graph should increse just as the TI cached_memory ‘with batch size incresed by one, memory increases about 5G’.<br>
So I’m so confused of this issue. Is it possible that pytorch will distribute more memory for the matrix with sufficent gpu memory in your compute? Thanks for your reply!</p>
<div class="md-table">
<table>
<thead>
<tr>
<th style="text-align:center">batch size</th>
<th style="text-align:center">TI allocated</th>
<th style="text-align:center">TI cached</th>
<th style="text-align:center">LoRA allocated +CTI</th>
<th style="text-align:center">LoRA cached+CTI</th>
<th>LoRA allocated -CTI</th>
<th>LoRA cached-CTI</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:center">4</td>
<td style="text-align:center">4674.74 MB</td>
<td style="text-align:center">25890.00 MB</td>
<td style="text-align:center">4866.56 MB</td>
<td style="text-align:center">25962.00 MB</td>
<td>4576.99 MB</td>
<td>25960.00 MB</td>
</tr>
<tr>
<td style="text-align:center">5</td>
<td style="text-align:center">4674.74 MB</td>
<td style="text-align:center">31212.00 MB</td>
<td style="text-align:center">4866.56 MB</td>
<td style="text-align:center">17792.00 MB</td>
<td>4576.99 MB</td>
<td>17492.00 MB</td>
</tr>
</tbody>
</table>
</div>

ptrblck: <aside class="quote no-group" data-username="Youguang_Xing" data-post="9" data-topic="56684">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/youguang_xing/48/58458_2.png" class="avatar"> Youguang_Xing:</div>
<blockquote>
<p>Is it possible that pytorch will distribute more memory for the matrix with sufficent gpu memory in your compute?</p>
</blockquote>
</aside>
<p>I don’t know what this means, but please refer to my previous post about workspaces used in e.g. <code>cudnn.benchmark</code>.<br>
It also still doesn’t make sense that the allocated memory stays the same so I guess you might not b emeasuring the inputs and created intermediates.</p>

Youguang_Xing: <p>Ok, thank you. I’ll try it later!</p>