A huge difference of memory usage on different GPU

username: wb-08
cooked: <p>I have two workstation: one with RTX 3090 and another with RTX A6000. The model on RTX 3090 takes 8897MB in total.When it runs on the RTX A6000 it takes 36945MB. I have identical cuda, cudnn and pytorch version</p>

username: ptrblck
cooked: <p>Is the memory allocated or only reserved in the cache?<br>
In the latter case, more available memory could be used for e.g. the cudnn workspace to check for faster kernels and could be reused afterwards.</p>

username: wb-08
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/b/ebca4f77df7c9fe967c1e0224cbac180e0eec61d.png" data-download-href="https://discuss.pytorch.org/uploads/default/ebca4f77df7c9fe967c1e0224cbac180e0eec61d" title="Снимок экрана от 2021-07-14 10-50-06"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/b/ebca4f77df7c9fe967c1e0224cbac180e0eec61d_2_508x500.png" alt="Снимок экрана от 2021-07-14 10-50-06" data-base62-sha1="xDTEwA707WtCUTOFflyZ8OonGyN" width="508" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/b/ebca4f77df7c9fe967c1e0224cbac180e0eec61d_2_508x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/b/ebca4f77df7c9fe967c1e0224cbac180e0eec61d_2_762x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/b/ebca4f77df7c9fe967c1e0224cbac180e0eec61d.png 2x" data-dominant-color="F2F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Снимок экрана от 2021-07-14 10-50-06</span><span class="informations">879×865 92.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> - torch.cuda.memory_summary()<br>
37039898624 - torch.cuda.memory_cached()<br>
37039898624 - torch.cuda.memory_reserved()</p>

username: ptrblck
cooked: <p>Based on the output, no memory is allocated (only in the cache), which is a bit strange.<br>
Did you delete all tensors before checking the memory stats?</p>

username: wb-08
cooked: <p>No, i must delete the object with del and then apply torch.cuda.empty_cache()?</p>

username: ptrblck
cooked: <p>No, you do not have to delete anything, if it’s still used, but allocated memory should have been shown in this case as seen here:</p>
<pre><code class="lang-python">import torch

print(torch.cuda.memory_summary())
|===========================================================================|
|                  PyTorch CUDA memory summary, device ID 0                 |
|---------------------------------------------------------------------------|
|            CUDA OOMs: 0            |        cudaMalloc retries: 0         |
|===========================================================================|
|        Metric         | Cur Usage  | Peak Usage | Tot Alloc  | Tot Freed  |
|---------------------------------------------------------------------------|
| Allocated memory      |       0 B  |       0 B  |       0 B  |       0 B  |
|       from large pool |       0 B  |       0 B  |       0 B  |       0 B  |
|       from small pool |       0 B  |       0 B  |       0 B  |       0 B  |
|---------------------------------------------------------------------------|
...

x = torch.randn(1024, device='cuda')
print(torch.cuda.memory_summary())
|===========================================================================|
|                  PyTorch CUDA memory summary, device ID 0                 |
|---------------------------------------------------------------------------|
|            CUDA OOMs: 0            |        cudaMalloc retries: 0         |
|===========================================================================|
|        Metric         | Cur Usage  | Peak Usage | Tot Alloc  | Tot Freed  |
|---------------------------------------------------------------------------|
| Allocated memory      |    4096 B  |    4096 B  |    4096 B  |       0 B  |
|       from large pool |       0 B  |       0 B  |       0 B  |       0 B  |
|       from small pool |    4096 B  |    4096 B  |    4096 B  |       0 B  |
|---------------------------------------------------------------------------|
| Active memory         |    4096 B  |    4096 B  |    4096 B  |       0 B  |
|       from large pool |       0 B  |       0 B  |       0 B  |       0 B  |
|       from small pool |    4096 B  |    4096 B  |    4096 B  |       0 B  |
|---------------------------------------------------------------------------|
| GPU reserved memory   |    2048 KB |    2048 KB |    2048 KB |       0 B  |
|       from large pool |       0 KB |       0 KB |       0 KB |       0 B  |
|       from small pool |    2048 KB |    2048 KB |    2048 KB |       0 B  |
|---------------------------------------------------------------------------|
| Non-releasable memory |    2044 KB |    2044 KB |    2044 KB |       0 B  |
|       from large pool |       0 KB |       0 KB |       0 KB |       0 B  |
|       from small pool |    2044 KB |    2044 KB |    2044 KB |       0 B  |
|---------------------------------------------------------------------------|
| Allocations           |       1    |       1    |       1    |       0    |
|       from large pool |       0    |       0    |       0    |       0    |
|       from small pool |       1    |       1    |       1    |       0    |
|---------------------------------------------------------------------------|
| Active allocs         |       1    |       1    |       1    |       0    |
|       from large pool |       0    |       0    |       0    |       0    |
|       from small pool |       1    |       1    |       1    |       0    |
|---------------------------------------------------------------------------|
| GPU reserved segments |       1    |       1    |       1    |       0    |
|       from large pool |       0    |       0    |       0    |       0    |
|       from small pool |       1    |       1    |       1    |       0    |
|---------------------------------------------------------------------------|
| Non-releasable allocs |       1    |       1    |       1    |       0    |
|       from large pool |       0    |       0    |       0    |       0    |
|       from small pool |       1    |       1    |       1    |       0    |
|===========================================================================|
</code></pre>
<p>Here you can see that the memory is initially empty and after creating a tensor with <code>1024</code> elements in <code>float32</code> <code>4096</code> bytes will be allocated and 2MB will be in the cache.</p>
<p>However, your currently allocated memory is zero, so no active tensors are on the GPU.</p>

username: wb-08
cooked: <p>Sorry, it’s right:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/d/dd9a1d424bfdaf223bd477aa066bc330b9d839a9.png" data-download-href="https://discuss.pytorch.org/uploads/default/dd9a1d424bfdaf223bd477aa066bc330b9d839a9" title="1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/d/dd9a1d424bfdaf223bd477aa066bc330b9d839a9_2_513x499.png" alt="1" data-base62-sha1="vCnHYCy69OfsOlzgUOHMLWT4H4R" width="513" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/d/dd9a1d424bfdaf223bd477aa066bc330b9d839a9_2_513x499.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/d/dd9a1d424bfdaf223bd477aa066bc330b9d839a9_2_769x748.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/d/dd9a1d424bfdaf223bd477aa066bc330b9d839a9.png 2x" data-dominant-color="F2F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1</span><span class="informations">894×871 94.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
But when i use with torch.no_grad():<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/2/e2b77f8eeda70183ef657edaa89e491f3b940c22.png" data-download-href="https://discuss.pytorch.org/uploads/default/e2b77f8eeda70183ef657edaa89e491f3b940c22" title="2"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/2/e2b77f8eeda70183ef657edaa89e491f3b940c22_2_485x500.png" alt="2" data-base62-sha1="wlD3ccCp3bXplJahGLx0QxnqoSK" width="485" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/2/e2b77f8eeda70183ef657edaa89e491f3b940c22_2_485x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/2/e2b77f8eeda70183ef657edaa89e491f3b940c22_2_727x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/2/e2b77f8eeda70183ef657edaa89e491f3b940c22.png 2x" data-dominant-color="F2F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">2</span><span class="informations">912×939 105 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: wb-08
cooked: <p>It seems to me that torch.backends.cudnn.benchmark = False solve my problem) Thank you!</p>