Inconsistent model inference Time

milesyang: <p>Hi all. I am a graduate student working on a project about Model inference time optimization.<br>
I am using this very simple code snippet to benchmark the inference time of resenet50:</p>
<pre><code class="lang-auto">    for i, (batch_input, batch_target) in enumerate(data_loader):
        batch_input_var = batch_input.to(device)
        torch.cuda.synchronize()


        # time.sleep(0.2)
        start = torch.cuda.Event(enable_timing=True)
        end = torch.cuda.Event(enable_timing=True)
        start.record()
        with torch.no_grad():
            output = net(batch_input_var)
        end.record()
        torch.cuda.synchronize()
        print(start.elapsed_time(end))
</code></pre>
<p>It gives me 10.5ms consistently.</p>
<p>However, if I uncomment time.sleep(0.2), or do something else which takes around 0.05s to 0.2s, the results start to be highly inconsistent, ranging from 12.7ms to 26.1ms. If I set the sleep time to say 2 seconds, the results become consistent again, around 15ms.</p>
<p>Please I am wondering if you have any thoughts on this?</p>
<p>Thanks!</p>

milesyang: <p>any hope to find help for this?</p>

ptrblck: <p>I haven’t seen this issue before and also cannot reproduce it using a small example model and random input data.<br>
Could you share a code snippet to reproduce this issue and also share your setup information?</p>

milesyang: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>   Thanks a lot for your response!</p>
<p>I have a minimal script to show this problem at the end.</p>
<p>I am using Python 3.6.9, Pytorch version 1.7.1, I’m using RTX 2080, CUDA version is 11.0, driver version 450.80.02</p>
<p>Script:</p>
<pre><code class="lang-auto">import torchvision.models as models
import torch
import numpy as np
import time


if __name__ == '__main__':
    device=0
    net = models.resnet50(pretrained=True).to(device)
    net.eval()

    data = np.random.uniform(0, 1, (200, 3, 480, 854))
    label = np.random.randint(0, 100, (200,))
    tensor_imgs = torch.Tensor(data.astype(np.float32))
    tensor_labels = torch.Tensor(label.astype(np.float32))
    dataset = torch.utils.data.dataset.TensorDataset(tensor_imgs, tensor_labels)
    data_loader = torch.utils.data.dataloader.DataLoader(dataset, batch_size=1, shuffle=False, num_workers=10)

    for i, (batch_input, batch_target) in enumerate(data_loader):
        if i &gt;= 100:
            break
        batch_input_var = batch_input.to(device)
        output = net(batch_input_var)
    print('Warm-up finished')


    for i, (batch_input, batch_target) in enumerate(data_loader):
        batch_input_var = batch_input.to(device)
        torch.cuda.synchronize()

        time.sleep(0.2)
        start = torch.cuda.Event(enable_timing=True)
        end = torch.cuda.Event(enable_timing=True)
        start.record()
        with torch.no_grad():
            output = net(batch_input_var)
        end.record()
        torch.cuda.synchronize()
        print(start.elapsed_time(end))
</code></pre>

ptrblck: <p>Thanks I can reproduce the different timing results using <code>time.sleep</code>.<br>
However, if I add a real workload, e.g.:</p>
<pre><code class="lang-python">def fun2():
    for _ in range(200):
        np.matmul(np.random.randn(1024, 1024), np.random.randn(1024, 1024))

if __name__ == '__main__':
    device=0
    ...
    for i, (batch_input, batch_target) in enumerate(data_loader):
        batch_input_var = batch_input.to(device)
        torch.cuda.synchronize()

        #time.sleep(0.2)
        fun2()
        ...
</code></pre>
<p>I get valid results again (although the code overall will have a large delay), so I assume <code>time</code> might distort the profiling somehow.<br>
I’ll ask around, since this is new to me.</p>

milesyang: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks!</p>
<p>The motivation I ask this is because I am designing a real time scheduler for model inference.</p>
<p>When there’s no workload in the system (system idle waiting for requests) and then an inference request comes and starts to be processed on GPU, it gives the unstable inference time, just like the situation of using time.sleep(). When GPU is busy processing some data and then immediately switches to new data,  the inference time of these new data is quite normal.</p>
<p>Also I tried time.perf_counter to replace torch.event,  it gives similar results.</p>
<p>I hope this provides more context information for this problem.</p>
<p>Thank you very much!</p>

ptrblck: <p>The “wake up” time of the GPU might be expected and I don’t know, if you can avoid it, since the GPU would reduce it’s <code>Perf</code> to use less power (as seen via <code>nvidia-smi</code>) if no workload is currently executed.<br>
What I don’t know, is why <code>time.sleep</code> interferes with the profiling, but I’m also not an expert in Python mutliprocessing and don’t know if the main process is just suspended for the specified amount of time or if “more” is happening under the hood.</p>

milesyang: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks for the direction!<br>
I will check if there’s a way to keep GPU alive.</p>

ptrblck: <p>You can check the setup of your devices via <code>nvidia-smi -q -d PERFORMANCE</code> and should see an active <code>Clocks Throttle Reason</code> as <code>Active</code> (most likely <code>Idle</code>).<br>
Using <code>nvidia-settings</code> you could try to force your device to run at <code>P0</code> at all times, if that’s what you desire.</p>

milesyang: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> I tried the command <code>nvidia-settings -a '[gpu:0]/GPUPowerMizerMode=1'</code>. It changes my Perf to P0 in nvidia-smi. But whenever I launch the script it falls back to P2 (throttle reasons are all inactive), and the inference time is still jumping around. When the script is finished it goes back to P0. Do you know if there’s a way to keep it at P0 please?</p>

feiyuhuahuo: <p>Hi <a class="mention" href="/u/milesyang">@milesyang</a>, do you solve this problem now?<br>
I test the code on RTX3090 (desktop) and RTX 2080 max-q (laptop). I found the Perf is always p2 for RTX3090 and the ineference time keeps the same however the sleep time is. But for RTX 2080 max-q, the Perf jumps with the sleep time. If sleep time is less than 50ms, the Perf is always P0 and the inference time is normal. But if sleep time is 500ms, the Perf jumps, maybe P0, P3 or P5, and the inference time jumps either. So I assume P2 is a normal power state for RTX3090 and P0 for RTX 2080 max-q. But since  RTX 2080 max-q is a laptop GPU, the Perf tends to jump.</p>