Are there any reasons why running GPU-inference in a thread would be slower?

Alexander_Soare: <p>Consider this code where I compare running a simple neural network in a thread vs directly (and I’ll highlight now: the thread is started then <em>immediately</em> joined):</p>
<pre data-code-wrap="python"><code class="lang-python">import threading
from time import perf_counter

import torch

DEVICE = torch.device("cuda")

class Foo:
    def __init__(self, model):
        self.model = model

    def foo(self, tensor):
        with torch.inference_mode():
            self.model(tensor)

    def run(self):
        # Warmup
        for _ in range(1000):
            tensor = torch.randn(size=(1000, 1000), device=DEVICE)
            self.foo(tensor)

        start = perf_counter()
        for _ in range(1000):
            tensor = torch.randn(size=(1000, 1000), device=DEVICE)
            self.thread = threading.Thread(target=self.foo, args=(tensor,))
            self.thread.start()
            self.thread.join()
        print("Loop with threads:", perf_counter() - start)

        start = perf_counter()
        for _ in range(1000):
            tensor = torch.randn(size=(1000, 1000), device=DEVICE)
            self.foo(tensor)
        print("Simple loop:", perf_counter() - start)

model = torch.nn.Sequential(*[torch.nn.Linear(1000, 1000) for _ in range(10)]).to(DEVICE)
foo = Foo(model)
foo.run()
</code></pre>
<p>I actually find that both loops run for the same amount of time. Great!</p>
<p>But, for my actual model, I get that the threaded version is 10x slower. One extra clue, is that if I run the inference on CPU, there is no relative slow down for the threaded version.</p>
<p>I’d like to get some clues as to why (and if not, I suppose I’ll have to give an example that’s truer to my <em>actual</em> model, which is not easy to do).</p>
<p>Btw my ultimate goal is to be running GPU-inference in order to populate a cache then be able to consume from that cache with CPU-bound work. In the meantime, GPU-inference runs again so that I can keep my cache from getting depleted, and never have to wait for inference.</p>

ptrblck: <aside class="quote no-group" data-username="Alexander_Soare" data-post="1" data-topic="204519">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alexander_soare/48/24720_2.png" class="avatar"> Alexander_Soare:</div>
<blockquote>
<p>I’d like to get some clues as to why (and if not, I suppose I’ll have to give an example that’s truer to my <em>actual</em> model, which is not easy to do).</p>
</blockquote>
</aside>
<p>Yes, this might be needed as your current code snippet isn’t able to reproduce the issue.<br>
In case it’s hard to isolate it down to a minimal code, you could try to profile your code to see which operations might cause the slowdown in both approaches.</p>

Alexander_Soare: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> thanks for picking this up! I dug deep into my model and found one of my submodules needed to be “warmed up”. I know of this phenomenon but have never been quite clear on it. It’s the idea that when you time inference you should do some warmup runs first before starting your timing loop because the first few runs take extra time.</p>
<p>So I discovered that these warmup effects are present within a thread as well. So even if you do</p>
<pre><code class="lang-auto"># Attempt to warm up.
for _ in range(10):
    run_inference()

# Warmup doesn't help here!
start = time.time()
for _ in range(10):
    thread = threading.Thread(target=run_inference)
    thread.start()
    thread.join()
print(time.time() - start)

# Warmup helps here!
start = time.time()
for _ in range(10):
    run_inference()
print(time.time() - start)
</code></pre>
<p>You still suffer warmup effects with each loop iteration for the threaded version. The solution is to have 1 thread for everything (which you can manage with queues). I was able to verify this works.</p>
<p>I’d love to get your take on this. Btw here is the block of code that I know goes slower the first time around: <a href="https://github.com/huggingface/lerobot/blob/125bd93e2975566eb55eebe3e0909aaf544f5ab2/lerobot/common/policies/diffusion/modeling_diffusion.py#L638-L641" rel="noopener nofollow ugc">here</a> where the <code>conv</code>s are defined <a href="https://github.com/huggingface/lerobot/blob/125bd93e2975566eb55eebe3e0909aaf544f5ab2/lerobot/common/policies/diffusion/modeling_diffusion.py#L691" rel="noopener nofollow ugc">here</a> and the <code>downsample</code> is defined <a href="https://github.com/huggingface/lerobot/blob/125bd93e2975566eb55eebe3e0909aaf544f5ab2/lerobot/common/policies/diffusion/modeling_diffusion.py#L576" rel="noopener nofollow ugc">here</a>.</p>

ptrblck: <p>Warmups are needed for proper profiling as you could otherwise profile the startup time after the device was set to IDLE and reduced its power usage.<br>
With that being said, the actual issue is also the general usage of host timers without proper synchronizations. CUDA operations are executed asynchronously. If you use host timers without synchronizing the GPU you would measure the dispatching and kernel launches in the best case and noise in the worst case.<br>
Since you are explicitly using multiple threads, you would need to define what exactly you are interested in profiling (the launches while the GPU is still busy or the GPU execution time).<br>
A visual profiler might help here as it would show all threads and the execution time.</p>