Access GPU memory usage in Pytorch

username: yang
cooked: <p>In Torch, we use   cutorch.getMemoryUsage(i)   to obtain the memory usage of the i-th GPU.</p>
<p>Is there a similar function in Pytorch?</p>

username: miguelvr
cooked: <p>You need that for your script? If so, I don’t know how. Otherwise, you can run <code>nvidia-smi</code> in the terminal to check that</p>

username: yang
cooked: <p>Yes, I try to use it in script.<br>
The goal is to automatically find a GPU with enough memory left.</p>
<pre><code class="lang-python">import torch.cuda as cutorch

for i in range(cutorch.device_count()):
     if cutorch.getMemoryUsage(i) &gt; MEM: 
         opts.gpuID = i
         break
</code></pre>

username: mjstevens777
cooked: <p>In case anyone else stumbles across this thread, I wrote a script to query nvidia-smi that might be helpful.</p>
<pre><code>import subprocess

def get_gpu_memory_map():
    """Get the current gpu usage.

    Returns
    -------
    usage: dict
        Keys are device ids as integers.
        Values are memory usage as integers in MB.
    """
    result = subprocess.check_output(
        [
            'nvidia-smi', '--query-gpu=memory.used',
            '--format=csv,nounits,noheader'
        ], encoding='utf-8')
    # Convert lines into a dictionary
    gpu_memory = [int(x) for x in result.strip().split('\n')]
    gpu_memory_map = dict(zip(range(len(gpu_memory)), gpu_memory))
    return gpu_memory_map</code></pre>

username: laoreja
cooked: <p>GPUtil is also a library that achieve the same goal.<br>
But I’m wondering if PyTorch has some functions for this purpose.<br>
My goal is to measure the exact memory usage of my model, and it varies as the input size varies, so I’m wondering if PyTorch could have such a function so that I can have more accurate GPU memory usage estimation.</p>

username: albanD
cooked: <p>Hi,</p>
<p>You can find in the doc these functions <a href="https://pytorch.org/docs/stable/cuda.html#torch.cuda.max_memory_allocated">here</a> and below. You can get the current and max memory allocated on the gpu and the current max memory used to actually store tensors.</p>

username: laoreja
cooked: <p>Hi, thank you!<br>
I found these functions later, but find that they did not match the nvidia-smi output. And what’s the difference between max_memory_allocated/cached V.S. memory_allocated/cached?</p>
<p>I raised these questions in <a href="https://discuss.pytorch.org/t/memory-cached-and-memory-allocated-does-not-nvidia-smi-result/28420">Memory_cached and memory_allocated does not nvidia-smi result</a></p>

username: pltrdy
cooked: <p>Just FYI, I was also looking for the total GPU memory, which can be found with:</p>
<pre><code class="lang-auto">torch.cuda.get_device_properties(device).total_memory
</code></pre>

username: Sherzod
cooked: <p>3 highly voted answers show different results. Why? Or am I missing something?</p>

username: ptrblck
cooked: <p>The different answers explain what the use case of the code snippet is, e.g. printing the information of <code>nvidia-smi</code> inside the script, checking the current and max. allocated memory, and printing the total memory of a specific device, so you can chose whatever fits your use case of “memory usage”. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>