It there anyway to let program select free GPU automatically?

username: Mactarvish
cooked: <p>Hi guys, I’ve got a two-GPUs PC and try to run two networks on GPUs parallelly. For this, now when I run one of them, I set <code>torch.cuda.set_device(0)</code> and <code>torch.cuda.set_device(1)</code> for another one. It there any functions or orders to judge which GPU is free and select it? Thank you very much~</p>

username: ptrblck
cooked: <p>I’m not aware of a straightforward way, so please correct me, if there is a simpler solution.<br>
For now you could call <code>nvidia-smi</code> inside your script and based on this output select the GPU you would like to use. The returned memory stats will be wrapped into a <code>pandas.DataFrame</code> so that you can sort it etc.<br>
Here is a sample code:</p>
<pre><code class="lang-auto">import subprocess
import sys
if sys.version_info[0] &lt; 3: 
    from StringIO import StringIO
else:
    from io import StringIO
import pandas as pd

def get_free_gpu():
    gpu_stats = subprocess.check_output(["nvidia-smi", "--format=csv", "--query-gpu=memory.used,memory.free"])
    gpu_df = pd.read_csv(StringIO(gpu_stats),
                         names=['memory.used', 'memory.free'],
                         skiprows=1)
    print('GPU usage:\n{}'.format(gpu_df))
    gpu_df['memory.free'] = gpu_df['memory.free'].map(lambda x: x.rstrip(' [MiB]'))
    idx = gpu_df['memory.free'].idxmax()
    print('Returning GPU{} with {} free MiB'.format(idx, gpu_df.iloc[idx]['memory.free']))
    return idx

free_gpu_id = get_free_gpu()
torch.cuda.set_device(free_gpu_id)
</code></pre>

username: Mactarvish
cooked: <p>Thanks for your reply, but I got such an error when I ran your code:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/home/hdl2/Desktop/SonoFetalImage/playground.py", line 21, in &lt;module&gt;
    free_gpu_id = get_free_gpu()
  File "/home/hdl2/Desktop/SonoFetalImage/playground.py", line 12, in get_free_gpu
    gpu_df = pd.read_csv(StringIO(gpu_status),
TypeError: initial_value must be str or None, not bytes
</code></pre>
<p>How to solve it?</p>

username: ptrblck
cooked: <p>This error is thrown by <code>StringIO</code> for example if you included <code>from io import StringIO</code> while using Python2.<br>
That’s strange, since I built in a version check.</p>
<p>Which Python version are you using?</p>

username: Mactarvish
cooked: <p>Python 3.6.0 |Anaconda custom (64-bit)| (default, Dec 23 2016, 12:22:00)<br>
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux</p>

username: ptrblck
cooked: <p>I’m not sure, why this error occurs, but it seems to be related to handling unicode strings.</p>
<p>I tried the code with Python3.6 and it worked by adding the following bit:</p>
<pre><code class="lang-auto">gpu_df = pd.read_csv(StringIO(u"".join(gpu_stats)),
                         names=['memory.used', 'memory.free'],
                         skiprows=1)
</code></pre>

username: Mactarvish
cooked: <p>Regardless of these, I seem to find a simpler way, just like that:</p>
<pre><code class="lang-auto">def get_freer_gpu():
    os.system('nvidia-smi -q -d Memory |grep -A4 GPU|grep Free &gt;tmp')
    memory_available = [int(x.split()[2]) for x in open('tmp', 'r').readlines()]
    return np.argmax(memory_available)
</code></pre>
<p><img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=5" title=":smile:" class="emoji" alt=":smile:"></p>

username: ptrblck
cooked: <p>Nice! Looks indeed a bit shorter <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=5" title=":wink:" class="emoji" alt=":wink:"></p>

username: yusanshi
cooked: <p>I think you can try this:</p><aside class="quote quote-modified" data-post="5" data-topic="24281">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yusanshi/48/33622_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-can-find-available-gpu/24281/5">How can find available gpu?</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    You can use this to figure out the GPU id with the most free memory: 
nvidia-smi --query-gpu=memory.free --format=csv,nounits,noheader | nl -v 0 | sort -nrk 2 | cut -f 1 | head -n 1 | xargs

So instead of: 
python3 train.py

You can use: 
CUDA_VISIBLE_DEVICES=$(nvidia-smi --query-gpu=memory.free --format=csv,nounits,noheader | nl -v 0 | sort -nrk 2 | cut -f 1 | head -n 1 | xargs) python3 train.py

You can also add it into your .bashrc then you can directly run python3 train.py: 
alias python3='C…
  </blockquote>
</aside>


username: pen_good
cooked: <pre><code class="lang-auto">import os
def find_gpus(nums=6):
    os.system('nvidia-smi -q -d Memory |grep -A4 GPU|grep Free &gt;tmp_free_gpus')
    with open('tmp_free_gpus', 'r') as lines_txt:
        frees = lines_txt.readlines()
        idx_freeMemory_pair = [ (idx,int(x.split()[2]))
                              for idx,x in enumerate(frees) ]
    idx_freeMemory_pair.sort(key=lambda my_tuple:my_tuple[1],reverse=True)
    usingGPUs = [str(idx_memory_pair[0])
                    for idx_memory_pair in idx_freeMemory_pair[:nums] ]
    usingGPUs =  ','.join(usingGPUs)
    print('using GPU idx: #', usingGPUs)
    return usingGPUs
os.environ['CUDA_VISIBLE_DEVICES'] = find_gpus(nums=4)  # 必须在import torch前面

</code></pre>

username: afspies
cooked: <p>Hi,</p>
<p>Here’s a version which doesn’t write to an intermediate file, and lets you set the vram threshold:</p>
<p><a href="https://gist.github.com/afspies/7e211b83ca5a8902849b05ded9a10696" rel="noopener nofollow ugc">Automatic GPU Allocation (github.com)</a></p>
<p>This is based on <a class="mention" href="/u/pen_good">@pen_good</a> and <a class="mention" href="/u/ptrblck">@ptrblck</a> 's versions - thanks!</p>
<p>Alex</p>

username: pen_good
cooked: <p>Today,  I find this command  does not give output anymore</p>
<p>‘nvidia-smi -q -d Memory |grep -A4 GPU|grep Free’</p>

username: Seonjun_Kim
cooked: <pre><code class="lang-auto">import subprocess

def run_cmd(cmd):
    out = (subprocess.check_output(cmd, shell=True)).decode('utf-8')[:-1]
    return out

def get_free_gpu_indices():
    out = run_cmd('nvidia-smi -q -d Memory | grep -A4 GPU')
    out = (out.split('\n'))[1:]
    out = [l for l in out if '--' not in l]

    total_gpu_num = int(len(out)/5)
    gpu_bus_ids = []
    for i in range(total_gpu_num):
        gpu_bus_ids.append([l.strip().split()[1] for l in out[i*5:i*5+1]][0])

    out = run_cmd('nvidia-smi --query-compute-apps=gpu_bus_id --format=csv')
    gpu_bus_ids_in_use = (out.split('\n'))[1:]
    gpu_ids_in_use = []

    for bus_id in gpu_bus_ids_in_use:
        gpu_ids_in_use.append(gpu_bus_ids.index(bus_id))

    return [i for i in range(total_gpu_num) if i not in gpu_ids_in_use]

print(get_free_gpu_indices())
</code></pre>
<p>This gives entire list of free GPUs available.</p>

username: SamerMakni
cooked: <p>Hello, I was browsing this trying to find the best way to deal with this, since I’m working with a shared workstation that has multiple devices. So I made this package that auto selects the optimal cuda device, based on available memory, utilization, or power usage.</p>
<p>To install:</p>
<pre><code class="lang-auto">pip install cuda-selector
</code></pre>
<p>To use:</p>
<pre><code class="lang-auto">from cuda_selector import auto_cuda

# Select cuda device with most memory available
device = auto_cuda()

# Select cuda device with lowest power usasge usage
device = auto_cuda('power')

# Select cuda device with lowest power usasge usage
device = auto_cuda('utilization')```</code></pre>