`memory_stats` reports extremely high memory usage. How to interpret?

ege-erdogan: <p>Printing <code>memory_stats</code>, I get the following output reporting 300+ GB of memory usage, but the code successfully runs and the system obviously does not have that much memory. How should I interpret this value and get an actual estimate of the memory required for my program?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/d/bd8256980e16dcc07ff5066329584ffd89d1a5cf.png" data-download-href="https://discuss.pytorch.org/uploads/default/bd8256980e16dcc07ff5066329584ffd89d1a5cf" title="Screenshot 2023-02-27 at 13.28.08"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/d/bd8256980e16dcc07ff5066329584ffd89d1a5cf.png" alt="Screenshot 2023-02-27 at 13.28.08" data-base62-sha1="r2tuoSIBwXOSHu2AdHGcWejboR1" width="690" height="478" data-dominant-color="222222"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2023-02-27 at 13.28.08</span><span class="informations">1116×774 41.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thanks in advance for any answers!</p>

ptrblck: <p>I assume you are referring to the total allocations and freed tensors?<br>
If so, note that these stats accumulate the memory allocations as seen in this small example:</p>
<pre><code class="lang-python">x = torch.randn(1024, device="cuda") # 4kB
print(torch.cuda.memory_summary())
# |===========================================================================|
# |                  PyTorch CUDA memory summary, device ID 0                 |
# |---------------------------------------------------------------------------|
# |            CUDA OOMs: 0            |        cudaMalloc retries: 0         |
# |===========================================================================|
# |        Metric         | Cur Usage  | Peak Usage | Tot Alloc  | Tot Freed  |
# |---------------------------------------------------------------------------|
# | Allocated memory      |   4096 B   |   4096 B   |   4096 B   |      0 B   |
# |       from large pool |      0 B   |      0 B   |      0 B   |      0 B   |
# |       from small pool |   4096 B   |   4096 B   |   4096 B   |      0 B   |
# |---------------------------------------------------------------------------|
# | Active memory         |   4096 B   |   4096 B   |   4096 B   |      0 B   |
# |       from large pool |      0 B   |      0 B   |      0 B   |      0 B   |
# |       from small pool |   4096 B   |   4096 B   |   4096 B   |      0 B   |
# |---------------------------------------------------------------------------|
# | Requested memory      |   4096 B   |   4096 B   |   4096 B   |      0 B   |
# |       from large pool |      0 B   |      0 B   |      0 B   |      0 B   |
# |       from small pool |   4096 B   |   4096 B   |   4096 B   |      0 B   |
# |---------------------------------------------------------------------------|
# | GPU reserved memory   |   2048 KiB |   2048 KiB |   2048 KiB |      0 B   |
# |       from large pool |      0 KiB |      0 KiB |      0 KiB |      0 B   |
# |       from small pool |   2048 KiB |   2048 KiB |   2048 KiB |      0 B   |
# |---------------------------------------------------------------------------|
# | Non-releasable memory |   2044 KiB |   2044 KiB |   2044 KiB |      0 B   |
# |       from large pool |      0 KiB |      0 KiB |      0 KiB |      0 B   |
# |       from small pool |   2044 KiB |   2044 KiB |   2044 KiB |      0 B   |
# |---------------------------------------------------------------------------|
# | Allocations           |       1    |       1    |       1    |       0    |
# |       from large pool |       0    |       0    |       0    |       0    |
# |       from small pool |       1    |       1    |       1    |       0    |
# |---------------------------------------------------------------------------|
# | Active allocs         |       1    |       1    |       1    |       0    |
# |       from large pool |       0    |       0    |       0    |       0    |
# |       from small pool |       1    |       1    |       1    |       0    |
# |---------------------------------------------------------------------------|
# | GPU reserved segments |       1    |       1    |       1    |       0    |
# |       from large pool |       0    |       0    |       0    |       0    |
# |       from small pool |       1    |       1    |       1    |       0    |
# |---------------------------------------------------------------------------|
# | Non-releasable allocs |       1    |       1    |       1    |       0    |
# |       from large pool |       0    |       0    |       0    |       0    |
# |       from small pool |       1    |       1    |       1    |       0    |
# |---------------------------------------------------------------------------|
# | Oversize allocations  |       0    |       0    |       0    |       0    |
# |---------------------------------------------------------------------------|
# | Oversize GPU segments |       0    |       0    |       0    |       0    |
# |===========================================================================|


for _ in range(1024):
    x = torch.randn(1024, device="cuda") 
print(torch.cuda.memory_summary())
# |===========================================================================|
# |                  PyTorch CUDA memory summary, device ID 0                 |
# |---------------------------------------------------------------------------|
# |            CUDA OOMs: 0            |        cudaMalloc retries: 0         |
# |===========================================================================|
# |        Metric         | Cur Usage  | Peak Usage | Tot Alloc  | Tot Freed  |
# |---------------------------------------------------------------------------|
# | Allocated memory      |   4096 B   |   8192 B   |   4100 KiB |   4096 KiB |
# |       from large pool |      0 B   |      0 B   |      0 KiB |      0 KiB |
# |       from small pool |   4096 B   |   8192 B   |   4100 KiB |   4096 KiB |
# |---------------------------------------------------------------------------|
# | Active memory         |   4096 B   |   8192 B   |   4100 KiB |   4096 KiB |
# |       from large pool |      0 B   |      0 B   |      0 KiB |      0 KiB |
# |       from small pool |   4096 B   |   8192 B   |   4100 KiB |   4096 KiB |
# |---------------------------------------------------------------------------|
# | Requested memory      |   4096 B   |   8192 B   |   4100 KiB |   4096 KiB |
# |       from large pool |      0 B   |      0 B   |      0 KiB |      0 KiB |
# |       from small pool |   4096 B   |   8192 B   |   4100 KiB |   4096 KiB |
# |---------------------------------------------------------------------------|
# | GPU reserved memory   |   2048 KiB |   2048 KiB |   2048 KiB |      0 B   |
# |       from large pool |      0 KiB |      0 KiB |      0 KiB |      0 B   |
# |       from small pool |   2048 KiB |   2048 KiB |   2048 KiB |      0 B   |
# |---------------------------------------------------------------------------|
# | Non-releasable memory |   2044 KiB |   2044 KiB |   6140 KiB |   4096 KiB |
# |       from large pool |      0 KiB |      0 KiB |      0 KiB |      0 KiB |
# |       from small pool |   2044 KiB |   2044 KiB |   6140 KiB |   4096 KiB |
# |---------------------------------------------------------------------------|
# | Allocations           |       1    |       2    |    1025    |    1024    |
# |       from large pool |       0    |       0    |       0    |       0    |
# |       from small pool |       1    |       2    |    1025    |    1024    |
# |---------------------------------------------------------------------------|
# | Active allocs         |       1    |       2    |    1025    |    1024    |
# |       from large pool |       0    |       0    |       0    |       0    |
# |       from small pool |       1    |       2    |    1025    |    1024    |
# |---------------------------------------------------------------------------|
# | GPU reserved segments |       1    |       1    |       1    |       0    |
# |       from large pool |       0    |       0    |       0    |       0    |
# |       from small pool |       1    |       1    |       1    |       0    |
# |---------------------------------------------------------------------------|
# | Non-releasable allocs |       1    |       2    |     513    |     512    |
# |       from large pool |       0    |       0    |       0    |       0    |
# |       from small pool |       1    |       2    |     513    |     512    |
# |---------------------------------------------------------------------------|
# | Oversize allocations  |       0    |       0    |       0    |       0    |
# |---------------------------------------------------------------------------|
# | Oversize GPU segments |       0    |       0    |       0    |       0    |
# |===========================================================================|
</code></pre>
<p>Your currently allocated memory can be seen in <code>Cur Usage - Allocated memory</code> and the cached memory of <code>Cur Usage - GPU reserved memory</code>.</p>