Efficient way to run independent jobs on multiple GPU machine

shaul: <p>Hello, it is unclear to me what is the efficient way to run independent jobs (e.g., the many multiple runs of a hyper-parameter search effort) on a machine with multiple GPUs. I am not sure how Pytorch handles multiple GPUs, but I can see three ways with each possibly being better depending on how multiple GPUs are handled:</p>
<ol>
<li>Run the jobs one by one serially on the machine</li>
<li>Run as many jobs in parallel on the machine, limiting each to a different GPU with CUDA_VISIBLE_DEVICES</li>
<li>Run as many jobs in parallel on the machine, without limiting runs to specific GPUs<br>
Which of these options would be best? Or perhaps something else entirely?</li>
</ol>

Yan_Wang: <p>Hi, does anyone have some ideas?</p>

richard: <p>Option 1 seems bad because you’re not exploiting the potential parallelism.</p>
<p>Option 3 is interesting. I’m not sure how pytorch picks which GPU to use if you don’t specify but I wouldn’t be surprised if everything ran on GPU 0 or something.</p>
<p>Option 2 (using CUDA_VISIBLE_DEVICES) looks good to me. Pytorch has this <code>setDevice</code> API where you can tell it what gpu to put each tensor on, but <a href="https://github.com/pytorch/pytorch/blob/67b2923a9dc829537a4f275c0875027722b015b5/torch/cuda/__init__.py#L195" rel="nofollow noopener">this code</a> seems to recommend just using CUDA_VISIBLE_DEVICES</p>

Liang: <p>I tried option 3 with <code>multiprocessing.Pool() + pool.apply_async() + gpu_list = multiprocessing.Manager().list([0]*10)</code>. I tried to use a list to record the GPU usages, e.g. one gpu at most runs two jobs. Add 1 on the gpu_list when starting the job and subtract 1 after finishing it. I <strong>succeeded</strong> after solving three major problems:</p>
<p>1, At the start of each job, they will read gpu_lock at the same time. More than 2 jobs may use be allocated to the same GPU.<br>
<strong>SOLUTION</strong>: use time.sleep(0.1) while submitting jobs to by <code>pool.apply_async()</code><br>
Or <strong>BETTER SOLUTION</strong>: use <code>multiprocessing.Lock()</code> to protect the <code>gpu_list</code>, add a lock when decreasing or adding <code>gpu_list</code>.</p>
<p>2, the pool would start the new processing immediately after finishing the old one, but the old GPU memory release is much slower, causing out-of-memory errors for the new process.<br>
<strong>SOLUTION</strong>: use time.sleep(30) to wait and then read gpu_list to select spare GPU.<br>
Or <strong>BETTER SOLUTION</strong>: use <code>del network + torch.cuda.empty_cache()</code> at the end of the process function to manually release gpu memory.</p>
<p>3, In my codes, it’s strange that I can run three jobs (3 GB each) by different main() at the same without any error. However, if I use multiprocessing, some processing will die due to out-of-memory. I check GPU usage and find that my jobs would suddenly use 10 GB and lead to the other two’s failure. But this situation never happens when I run three scripts separately. I still don’t know why.</p>
<p>Anyway, it’s very tricky to use multiprocessing to wrap GPU jobs. The errors are strange and most of time, there is no error at all (You need use pool.apply without recall function to debug). The pool would finish its job even some of them fail. By the way, don’t use <code>torch.cuda.is_available</code> or similar even in the main(), strange error occurs.</p>

RoshanRane: <p>Thank you for such a detailed report! Saved me hrs of debugging.</p>