Model inference takes 1 s! every 8 passes, otherwise <1 ms

username: Lisa_Koenigsberg
cooked: <p>Hello !! Please help. Thank you.</p>
<p>Inference is &lt;1 ms but every 8 times or so 1000 ms! I’m using pytorch 2.0.1 and have tried many things so far as you can see in the code. I’m on a windows computer and this happens on 2 computers.</p>
<p>here is my code:</p>
<p>class SegModel():<br>
def <strong>init</strong>(self, parameters):<br>
gc.collect()<br>
torch.cuda.empty_cache()<br>
os.environ[‘CUDA_VISIBLE_DEVICES’] = ‘0’<br>
ENCODER = “vgg19_bn”</p>
<h1><a name="encoder_weights-imagenet-1" class="anchor" href="#encoder_weights-imagenet-1"></a>ENCODER_WEIGHTS = ‘imagenet’</h1>
<p>CLASSES = [‘my class’]<br>
ACTIVATION = ‘sigmoid’ # could be None for logits or ‘softmax2d’ for multiclass segmentation<br>
DEVICE = ‘cuda’<br>
in_channels = 3</p>
<pre><code class="lang-auto">    # create segmentation model with pretrained encoder
    model = seg.UnetPlusPlus(
        encoder_name=ENCODER,
        # encoder_weights=ENCODER_WEIGHTS,
        classes=len(CLASSES),
        activation=ACTIVATION,
        in_channels=in_channels,

    )
    torch.backends.cudnn.benchmark = True
    torch.backends.cudnn.enabled = False
    torch.backends.cudnn.deterministic = True
    torch.set_flush_denormal(True)
    torch.no_grad()
    torch.set_grad_enabled(False)
    torch.jit.enable_onednn_fusion(True)
    ENCODER_WEIGHTS = utils.get_state_dict(parameters.path_to_weights)
    model.load_state_dict(ENCODER_WEIGHTS)
    model = model.half()
    model.eval()
    self.model = model.to(DEVICE)
    self.batch_size = 1        
    torch.cuda.synchronize()
    t0 = int(time())
    self.model = self.warmup_operation(40)
    torch.cuda.synchronize()
    t1 = int(time())
    print('total_warmup_time', (t1 - t0) * 1000)


def warmup_operation(self, n_warmup_interations):
    # gc.collect()
    # torch.cuda.empty_cache()
    dummy = torch.ones((self.batch_size, 3, 256, 256),dtype=torch.float16, device='cuda')/0.5
    traced_model = torch.jit.trace(self.model, dummy)
    traced_model = torch.jit.freeze(traced_model)
    # n_warmup_interations = 40
    for iter in range(n_warmup_interations):
        # torch.cuda.empty_cache()
        torch.cuda.synchronize()
        t0 = int(time())
        traced_model(dummy)
        torch.cuda.synchronize()
        t1 = int(time())
        print('warmup_time', (t1 - t0) * 1000)
    # # self.model(dummy)
    #
    return traced_model
</code></pre>
<p>when I instantiate the model above it runs the warmup_operation method and prints out the times in ms below:</p>
<p>warmup_time 0<br>
warmup_time 0<br>
warmup_time 0<br>
warmup_time 0<br>
warmup_time 1000<br>
warmup_time 0<br>
warmup_time 0<br>
warmup_time 0<br>
warmup_time 0<br>
warmup_time 0<br>
warmup_time 0<br>
warmup_time 0<br>
warmup_time 0<br>
warmup_time 1000<br>
warmup_time 0<br>
warmup_time 0<br>
warmup_time 0<br>
warmup_time 0<br>
warmup_time 0<br>
warmup_time 0<br>
warmup_time 0<br>
warmup_time 1000<br>
warmup_time 0<br>
warmup_time 0<br>
warmup_time 0<br>
warmup_time 0<br>
warmup_time 0<br>
warmup_time 0<br>
warmup_time 0<br>
warmup_time 1000<br>
warmup_time 0<br>
warmup_time 0<br>
warmup_time 0<br>
warmup_time 0<br>
warmup_time 0<br>
warmup_time 0<br>
warmup_time 0<br>
warmup_time 1000<br>
warmup_time 0<br>
warmup_time 0<br>
total_warmup_time 9000</p>
<p>Process finished with exit code 0</p>
<p>The above happens when doing inference with the model for real images.<br>
This is a huge mystery to me and I need to solve it since I have a real time application.</p>
<p>Thank you !!</p>
<p>Lisa Koenigsberg</p>

username: ptrblck
cooked: <p>Remove the rounding by dropping the <code>int()</code> call around the reporting, as the actual execution time might be in ms:</p>
<pre><code class="lang-python">for _ in range(10):
    t0_float = time.time()
    t0 = int(t0_float)
    
    time.sleep(1e-1)
    
    t1_float = time.time()
    t1 = int(t1_float)
    
    print(t1_float - t0_float)
    print(t1 - t0)
</code></pre>
<p>Output:</p>
<pre><code class="lang-python">0.10011005401611328
0
0.10011029243469238
0
0.10013604164123535
1
0.10013532638549805
0
0.10013484954833984
0
0.10013461112976074
0
0.10011577606201172
0
0.1001133918762207
0
0.10011577606201172
0
0.1001138687133789
0
</code></pre>

username: Lisa_Koenigsberg
cooked: <p>Yes, that certainly is it ! Thank you very much.<br>
Now that I fixed that, on my computer with a batch_size of 8 it takes 125 ms per pass which is too long for my application. Besides using a lighter model do you have any tips for what to try ?</p>
<p>Thank you !</p>

username: ptrblck
cooked: <p>You could profile your code to narrow down the bottleneck on the workflow.<br>
Based on this you could check what to optimize. E.g. if your use case if CPU-limited you might want to apply CUDAGraphs as described <a href="https://pytorch.org/docs/stable/notes/cuda.html#cuda-graphs">here</a>. Also, you could try the latest <code>torch.compile</code> util. (also with CUDAGraphs if needed) and check if kernel fusion and other optimizations could help.</p>