Identifying if gpu cores are used

username: Stephane_Ancelot
cooked: <p>Hi,</p>
<p>I have got the setup described in the image.</p>
<p>Either on windows or wsl2. altough the cuda is available, I don’t manage to run llm inference on gpu.<br>
It looks like the models are loaded on gpu memory, but It seems using cpu for inference.</p>
<p>Is there a way to monitor gpu activity ?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/8/e89b25fc8d05172dff54f38fd989c9bea1b386ab.png" data-download-href="https://discuss.pytorch.org/uploads/default/e89b25fc8d05172dff54f38fd989c9bea1b386ab" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/8/e89b25fc8d05172dff54f38fd989c9bea1b386ab_2_689x367.png" alt="image" data-base62-sha1="xbJaYubJzybrwtfr4wGrYe5KDa3" width="689" height="367" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/8/e89b25fc8d05172dff54f38fd989c9bea1b386ab_2_689x367.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/8/e89b25fc8d05172dff54f38fd989c9bea1b386ab_2_1033x550.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/8/e89b25fc8d05172dff54f38fd989c9bea1b386ab.png 2x" data-dominant-color="F3F6F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1223×652 60.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/a/5a2039a170b2e17d2a6e4589b75f3fe140c80e07.png" data-download-href="https://discuss.pytorch.org/uploads/default/5a2039a170b2e17d2a6e4589b75f3fe140c80e07" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/a/5a2039a170b2e17d2a6e4589b75f3fe140c80e07.png" alt="image" data-base62-sha1="cRi0T7OBSaYohBkEoX0WMnbsJP9" width="663" height="349"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">663×349 7.29 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I setted up the pip with extra url cuda torch version</p>

username: ptrblck
cooked: <p><code>nvidia-smi</code> should show the activity. Move a large tensor to the device and execute a <code>matmul</code> in a loop after you confirmed the tensor was moved to the GPU by checking its <code>.device</code> attribute as a quick check.</p>

username: python
cooked: <p>Yeah, it’s possible for a model to be loaded onto the GPU (so memory usage shows up) but still have inference happen on the CPU if the inputs or some parts of the model aren’t on the GPU.</p>
<p>To monitor actual GPU usage, you can use:</p>
<ul>
<li><strong>Windows</strong>: Task Manager &gt; Performance tab &gt; GPU section, but this sometimes doesn’t show detailed compute usage.</li>
<li><strong>WSL2 or Linux</strong>: Run <code>nvidia-smi</code> in the terminal. It shows GPU memory usage, running processes, and GPU utilization (look at the “Volatile GPU-Util” column).</li>
<li>For a more detailed view, try <code>nvidia-smi dmon</code> or use tools like <code>nvtop</code>.</li>
</ul>
<p>Also, make sure both your model and your inputs are on the same device (<code>cuda</code>). A common issue is forgetting to move the input tensors to the GPU before inference:</p>
<p>model = model.to(“cuda”)<br>
input = input.to(“cuda”)<br>
output = model(input)</p>
<p>If anything’s still unclear, feel free to share a code snippet; it might help track it down.</p>