Torch uses wrong GPU

username: Lue-C
cooked: <p>I am using a laptop with two GPUs. The first one is a Intel(R) UHD Graphics and the second one is a NVIDIA GeForce RTX 4090 Laptop GPU. I want to run a RAG application using a llama3 model using the second GPU. When I run it, the model is loaded into memory. Afterwards the second GPU is used only until the streaming of the output starts. When streaming only the first GPU is used. This can be seen by running the application and watching the performance with the task manager.</p>
<p>The problem can be reproduced with the following code:</p>
<pre><code class="lang-auto">import torch
import os
from transformers import AutoModelForCausalLM, AutoTokenizer, TextIteratorStreamer
from pathlib import Path

torch.set_default_device("cuda")

dir_ = Path(__file__).parent

model_path = "my_path"

tokenizer = AutoTokenizer.from_pretrained(model_path, device_map="cuda")
model = AutoModelForCausalLM.from_pretrained(
        model_path,
        torch_dtype=torch.bfloat16,
        device_map="cuda",
        load_in_8bit=True
)
model.generation_config.pad_token_id = tokenizer.pad_token_id

terminators = [
    tokenizer.eos_token_id,
    tokenizer.convert_tokens_to_ids("&lt;|eot_id|&gt;")
]

def respond(message):

    messages = []
    user_message = {"role": "user", "content":
        """
        Frage: '{query}'
        """.format(query=message)}
    messages.extend([{"role": "system", "content": 'Du bist ein Assistent, der Fragen beantwortet.'},
                        ]
                    )
    messages.append(user_message)

    text = tokenizer.apply_chat_template(
          messages,
          tokenize=False,
          add_generation_prompt=True
          )

    model_inputs = tokenizer([text], return_tensors="pt").to('cuda:0')

    generated_ids = model.generate(
          model_inputs.input_ids,
          do_sample=True,
          temperature=0.01,
          top_p=0.9
          )
    
    answer = tokenizer.decode(generated_ids[0], skip_special_tokens=True)
    answer = answer.split('assistant')[-1]
    return answer
    
print(respond('Welche Sehenswürdigkeiten gibt es in Berlin?'))
</code></pre>
<p>I checked that Cuda is using the right device by typing <code>python -m torch.utils.collect_env</code> in the console. This resulted in:</p>
<pre><code class="lang-auto">&lt;frozen runpy&gt;:128: RuntimeWarning: 'torch.utils.collect_env' found in sys.modules after import of package 'torch.utils', but prior to execution of 'torch.utils.collect_env'; this may result in unpredictable behaviour
Collecting environment information...
PyTorch version: 2.4.0+cu124
Is debug build: False
CUDA used to build PyTorch: 12.4
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 11 Pro
GCC version: Could not collect
Clang version: Could not collect
CMake version: Could not collect
Libc version: N/A

Python version: 3.11.7 (tags/v3.11.7:fa7a6f2, Dec  4 2023, 19:24:49) [MSC v.1937 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.22631-SP0
Is CUDA available: True
CUDA runtime version: 12.4.131
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 4090 Laptop GPU
Nvidia driver version: 551.78
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture=9
CurrentClockSpeed=2200
DeviceID=CPU0
Family=207
L2CacheSize=32768
L2CacheSpeed=
Manufacturer=GenuineIntel
MaxClockSpeed=2200
Name=13th Gen Intel(R) Core(TM) i9-13900HX
ProcessorType=3
Revision=

Versions of relevant libraries:
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.26.3
[pip3] onnxruntime==1.16.3
[pip3] torch==2.4.0+cu124
[pip3] torchaudio==2.4.0+cu124
[pip3] torchvision==0.19.0+cu124
[conda] Could not collect
</code></pre>
<p>I also tried</p>
<pre><code class="lang-auto">print(torch.cuda.is_available())
print(torch.cuda.device_count())
print(torch.cuda.current_device())
print(torch.cuda.get_device_name(0))
</code></pre>
<p>and got</p>
<pre><code class="lang-auto">True
1
0
'NVIDIA GeForce RTX 4090 Laptop GPU'
</code></pre>
<p>As far as I understand, everything runs with cuda and since the NVIDIA GPU is the only cuda device, only this GPU should be used.</p>
<p>Does somebody know how to use the NVIDIA GPU only when running the application? Why is the Intel GPU used when it is not even detected by cuda?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Lue-C" data-post="1" data-topic="208011">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lue-c/48/72085_2.png" class="avatar"> Lue-C:</div>
<blockquote>
<p>Afterwards the second GPU is used only until the streaming of the output starts. When streaming only the first GPU is used.</p>
</blockquote>
</aside>
<p>I doubt it as your Intel UHD Graphics seems to be an integrated GPU, which does not support CUDA (and I don’t know if any other Intel backends are supported on it).</p>
<aside class="quote no-group" data-username="Lue-C" data-post="1" data-topic="208011">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lue-c/48/72085_2.png" class="avatar"> Lue-C:</div>
<blockquote>
<p>This can be seen by running the application and watching the performance with the task manager.</p>
</blockquote>
</aside>
<p>Make sure to check the <code>compute</code> tab in your Task Manager or use <code>nvidia-smi</code> to see the utilization of your GPU.</p>

username: Lue-C
cooked: <p>Thanks for the advice! I checked the GPU utilization with nvidia-smi and it is used indeed with performance levels P2 to P5 and a utilization of ~15% to ~50%.</p>
<p>This is what I see in the task manager:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/5/850d6072c0d6a1fffb4686e764a1116cbcd99b6e.png" data-download-href="https://discuss.pytorch.org/uploads/default/850d6072c0d6a1fffb4686e764a1116cbcd99b6e" title="taskmtemp1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/5/850d6072c0d6a1fffb4686e764a1116cbcd99b6e_2_690x155.png" alt="taskmtemp1" data-base62-sha1="iZ2a73COdyuYf9P9DSu8VINynue" width="690" height="155" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/5/850d6072c0d6a1fffb4686e764a1116cbcd99b6e_2_690x155.png, https://discuss.pytorch.org/uploads/default/original/3X/8/5/850d6072c0d6a1fffb4686e764a1116cbcd99b6e.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/5/850d6072c0d6a1fffb4686e764a1116cbcd99b6e.png 2x" data-dominant-color="212121"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">taskmtemp1</span><span class="informations">934×211 57.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>This implies that the code above only utilizes the NVIDIA GPU, does it?<br>
Or did I miss something and the integrated GPU is slowing down the code?</p>

username: ptrblck
cooked: <p>Your integrated GPU doesn’t matter and won’t be used in PyTorch as it isn’t even detected. Your laptop itself will use it for visualization.</p>

username: Lue-C
cooked: <p>Thank you, this makes sense.<br>
But with this, the above code takes 73.22s to complete and return an answer. Is this a reasonable time for a response?</p>