High variance in timings across executions

Achkan_Salehi: <p>I’ve been trying to evaluate the performance of a few networks, and have not been unable to get consistent timings across executions, even for very small networks or just single convolution blocks. I am aware of the need to call <code>torch.cuda.synchronize</code>, and for good measure, I have set <code>torch.backends.cudnn.benchmark=True</code>. If that is relevant, I am using an <code>nvidia 1080ti</code> under Ubuntu <code>18.04</code> and the driver version that I’m using is <code>396.37</code>.</p>
<p>Consider this simple code, which corresponds to timing a single convolution:</p>
<pre><code class="lang-auto">import torch
import time
import matplotlib.pyplot as plt
import os 

os.environ["CUDA_VISIBLE_DEVICES"] = "1"
torch.backends.cudnn.benchmark = True

seed=0
torch.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
torch.backends.cudnn.deterministic = True

if __name__=="__main__":
   
    in_im=torch.rand(1,3,416,416).cuda()
   
    cnv= torch.nn.Conv2d(
            in_channels=3,
            out_channels=128,
            kernel_size=3,
            stride=1,
            padding=1,
            bias=True).cuda()
 
    skip=10
    num_runs=1000
    timings=[]
    for i in range(num_runs):
        torch.cuda.synchronize()
        t1 = time.time()
        _=cnv(in_im)
        torch.cuda.synchronize()
        t2 = time.time()
        if i&gt;=skip:
            timings.append(t2-t1)

    plt.plot(timings);
    plt.xlabel("execution number")
    plt.ylabel("time (s)")
    plt.xlim([skip,num_runs-skip])
    plt.title("Timing for single convolution block")
    plt.show()

</code></pre>
<p>I’m skipping the first <code>10</code> executions because the first few forward passes seem to always be slower (I assume because of memory transfers?). Here’s is the plot of the times I’m getting:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/2/234c8c000edc6f1f469a7ae1369a7d1a563f6cba.png" data-download-href="https://discuss.pytorch.org/uploads/default/234c8c000edc6f1f469a7ae1369a7d1a563f6cba" title="timing_info"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/2/234c8c000edc6f1f469a7ae1369a7d1a563f6cba_2_690x369.png" alt="timing_info" data-base62-sha1="52gIaqHRwstZRGndZzI78tqbU2S" width="690" height="369" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/2/234c8c000edc6f1f469a7ae1369a7d1a563f6cba_2_690x369.png, https://discuss.pytorch.org/uploads/default/optimized/2X/2/234c8c000edc6f1f469a7ae1369a7d1a563f6cba_2_1035x553.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/2/234c8c000edc6f1f469a7ae1369a7d1a563f6cba_2_1380x738.png 2x" data-dominant-color="F9FBFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">timing_info</span><span class="informations">1853×991 98.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>As you can see, the std deviation is relatively high (about 3.63e-05s), which might not seem like much, but it accumulates such that in real-world networks (<em>e.g.</em> YOLOv2) I’m getting a much higher variance, which makes it difficult to report a single FPS value for those networks. So my question is, is that normal behavior or am I missing a step for correct timing?</p>
<p>Thank you very much for your help, I appreciate it.</p>