Use two GPUs in C++

username: GabrielB
cooked: <p>Hello, I am trying the C++ frontend with <code>.to(torch::Device(torch::kCUDA, 1))</code> to load all my data and models onto the second GPU. The task manager shows the memory of the second GPU going up but the first GPU’s activity going up. How could it be processing the data on the wrong GPU? The same code works fine in Python. Can someone give an example of loading a model onto the second GPU so I can test it to see if it’s working properly?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="rtwolfe94022" data-post="2" data-topic="194567">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rtwolfe94022/48/67286_2.png" class="avatar"> rtwolfe94022:</div>
<blockquote>
<p>it might be related to CUDA context initialization or other operations that PyTorch performs on the default device before your instructions take effect.</p>
</blockquote>
</aside>
<p>That’s not the case as PyTorch will not initialize a CUDA context on the default device. This would also be visible in an increase in its memory usage.</p>
<aside class="quote no-group" data-username="rtwolfe94022" data-post="2" data-topic="194567">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rtwolfe94022/48/67286_2.png" class="avatar"> rtwolfe94022:</div>
<blockquote>
<p>and that the second GPU is visible and accessible by your application. You can check this by running <code>nvidia-smi</code> in your terminal.</p>
</blockquote>
</aside>
<p><code>nvidia-smi</code> will not tell you anything about the visibility of devices to processes.<br>
<a class="mention" href="/u/gabrielb">@GabrielB</a> you can mask a device via <code>CUDA_VISIBLE_DEVICES</code> to mask devices if needed.</p>
<aside class="quote no-group" data-username="rtwolfe94022" data-post="2" data-topic="194567">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rtwolfe94022/48/67286_2.png" class="avatar"> rtwolfe94022:</div>
<blockquote>
<p>by checking their <code>.device()</code> property.</p>
</blockquote>
</aside>
<p>NIT: <code>.device</code> is an attribute and not a function.</p>
<aside class="quote no-group" data-username="rtwolfe94022" data-post="2" data-topic="194567">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rtwolfe94022/48/67286_2.png" class="avatar"> rtwolfe94022:</div>
<blockquote>
<p>CUDA initializes its context on the first accessed GPU, which might explain the activity on the first GPU. This is a common behavior and doesn’t necessarily indicate that computation is happening on the wrong GPU.</p>
</blockquote>
</aside>
<p>This is not common at all and should be considered a bug.</p>
<aside class="quote no-group" data-username="rtwolfe94022" data-post="2" data-topic="194567">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rtwolfe94022/48/67286_2.png" class="avatar"> rtwolfe94022:</div>
<blockquote>
<p>PyTorch performs certain optimizations and background operations that might involve the GPUs. This activity might not directly relate to your data processing or model computation.</p>
</blockquote>
</aside>
<p>Could you describe what exactly you mean by that?</p>
<p><a class="mention" href="/u/rtwolfe94022">@rtwolfe94022</a> as previously asked, could you please review your ChatGPT posts before answering?<br>
Otherwise the text sounds plausible, but is wrong as seen in this example.</p>