Pytorch GPU question

n4tman: <p>Hi All,</p>
<p>Just wanted to ask</p>
<p>I do <code>device=torch.device('cuda' if torch.cuda.is_available() else 'cpu')</code> <code>or device=torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')</code> and then just move my data, tensors I want to use and my model to the device by doing .to(device) I am only using 1 GPU?</p>
<p>Thanks!</p>

ptrblck: <p>Yes, both approaches would use a single GPU only.</p>

n4tman: <p>Thanks. Would this still be the case if I had multiple GPUs available to pytorch?</p>

ptrblck: <p>Yes, this would still be the case and you could use <code>nn.DataParallel</code> or <code>nn.DistributedDataParallel</code> to use multiple devices.<br>
<a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html">This tutorial</a> gives you more information.</p>

n4tman: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>. Assuming I have at least 1 GPU, by default would the same GPU be used in both approaches?</p>

ptrblck: <p>If your system has only a single GPU, I think both approaches should fall back to using the single device.</p>

n4tman: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>. (referring to my original question) Just to confirm would the 2 approaches I specified at the top of the thread only use 1 GPU even if torch.cuda.device_count() gave a number greater than 1?</p>

ptrblck: <p>Yes, if you are explicitly moving the model and data to a specific device, only this device will be used.<br>
That will be the case as long as you don’t use e.g. <code>nn.DataParallel</code>.<br>
Note that (depending on your code) PyTorch might create a CUDA context on other visible devices.<br>
If you see this behavior and want to avoid it, you could mask the desired device via <code>CUDA_VISIBLE_DEVICES=0 python script.py args</code>.</p>

n4tman: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>  (again referring to my original question) if I have multiple GPUs on the system I execute my code, the 2 approches only use 1 GPU?</p>
<p>Also what exactly does a CUDA context mean and is is something I should avoid?</p>

ptrblck: <aside class="quote no-group" data-username="n4tman" data-post="12" data-topic="93260">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/bc8723/48.png" class="avatar"> n4tman:</div>
<blockquote>
<p>if I have multiple GPUs on the system I execute my code, the 2 approches only use 1 GPU?</p>
</blockquote>
</aside>
<p>Yes.</p>
<aside class="quote no-group" data-username="n4tman" data-post="12" data-topic="93260">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/bc8723/48.png" class="avatar"> n4tman:</div>
<blockquote>
<p>Also what exactly does a CUDA context mean and is is something I should avoid?</p>
</blockquote>
</aside>
<p>The CUDA context stores the GPU kernels, runtime etc., and thus uses memory on the specified device.</p>

n4tman: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>. Should cuda contexts be avoided?</p>

ptrblck: <p>No, as it’s holding the CUDA kernels. If you don’t initialize it, you won’t be able to run code of your GPU.</p>

n4tman: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>. So having cuda contexts on GPUs I don’t move anything to are fine?</p>