Inference of two models isn't parallelized on same GPU

YaraAlnaggar: <p>Hello,</p>
<p>I have a model that combines the inference of two models, as the following:</p>
<pre><code class="lang-auto">class model_all(nn.Module):
    def __init__(self,dense,shuffle):
        super(model_all,self).__init__()
        self.dense = dense
        self.shuffle = shuffle

    def forward(self,input):
        self.dense(input)
        self.shuffle(input)
</code></pre>
<p>I am expecting the inference of the two models to happen in a parallel since cuda is running asynchronously. In other words,</p>
<blockquote>
<p>model_all_inference_time &lt; (dense_inference_time + shuffle_infernce_time).</p>
</blockquote>
<p>However, what I get is</p>
<blockquote>
<p>model_all inference_time ~= (dense_inference_time + shuffle_infernce_time)</p>
</blockquote>
<p>When I measured inference time of model_all using this block of code output was 0.0228.</p>
<pre><code class="lang-auto">dummy = torch.ones((1,3,256,512)).to(device)
t = []
for _ in range(1000):
    tic = time.perf_counter()
    model(dummy)
    torch.cuda.synchronize()
    toc = time.perf_counter()
    t.append(toc-tic)
print(sum(t)/1000)
</code></pre>
<p>Then I measured  <code>self.shuffle</code> inference time by commenting <code>self.dense(input)</code> in forward function and output was 0.0069. Next, I measured  <code>self.dense</code> inference time  by commenting <code>self.shuffle(input)</code> and output was 0.0161. So, we can see that 0.0161+0.0069~=0.0228 which shows that models inference wasn’t in parallel but sequential .<br>
Also, GPU utilization doesn’t exceed 50% with model_all inference. But with self.dense infernce it reaches 65%.</p>
<p>Is there a way to run the sub two models in parallel?</p>
<p>I am using pytorch 1.1.0, cuda 9 and nvidia TITAN Xp.<br>
You can use this minimal code to reproduce the issue. Thank you!</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import time
import torchvision
device = torch.device('cuda')

class model_all(nn.Module):
    def __init__(self,dense,shuffle):
        super(model_all,self).__init__()
        self.dense = dense
        self.shuffle = shuffle
    def forward(self,input):
        self.dense(input)
        self.shuffle(input)

dense = torchvision.models.densenet121(pretrained=False).to(device)
shuffle = torchvision.models.shufflenet_v2_x0_5(pretrained=False).to(device)
model = model_all(dense,shuffle)
dummy = torch.ones((1,3,256,512)).to(device)
t = []
for _ in range(1000):
    tic = time.perf_counter()
    model(dummy)
    torch.cuda.synchronize()
    toc = time.perf_counter()
    t.append(toc-tic)
print(sum(t)/1000)
</code></pre>

ptrblck: <p>If a submodule already uses all SMs on your GPU, you won’t be able to run another model in parallel.<br>
A lot of layers are used in such a kernel launch configuration to saturate the GPU as much as possible.</p>

googlebot: <aside class="quote no-group" data-username="YaraAlnaggar" data-post="1" data-topic="81908">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yaraalnaggar/48/20024_2.png" class="avatar"> YaraAlnaggar:</div>
<blockquote>
<p>I am expecting the inference of the two models to happen in a parallel since cuda is running asynchronously.</p>
</blockquote>
</aside>
<p>You’re not using torch.cuda.Stream, this means that single serialized queue of cuda operations is used. So, even if your first model is non-blocking (memory copying is blocking), async mode of cuda by itself only allows you to enqueue ops from second model earlier.</p>