Why time.time() in python is inaccurte?

username: LeeDoYup
cooked: <p>I think it is an elementary question about programming with GPU.</p>
<p>First, i tried to use time.time() in python module, to measure the operation time of some modules in NNs.<br>
such as</p>
<pre><code class="lang-auto">def forward(self, x):
  end = time.time()
  output1 = self.layer1(x)
  time_output1 = time.time()
  output2 = self.layer2(output1)
  time_output2 = time.time()
  print(time_output1 - end, time_output2-end)
</code></pre>
<p>However, i found that the time information is inaccurate, and i have to use below link:</p><aside class="quote" data-post="1" data-topic="26964">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/daulbaev/48/1824_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-measure-time-in-pytorch/26964">How to measure time in PyTorch</a> 
  </div>
  <blockquote>
    I have seen lots of ways to measure time in PyTorch. But what is the most proper way to do it now (both for cpu and cuda)? 
Should I clear the memory cache if I use timeit? 
And is it possible to get accurate results if I’m computing on a cluster? And is it a way to make this results reproducible? 
And what is better: timeit or profiler?
  </blockquote>
</aside>

<p>Although i measure the correct time by the methods in the link,<br>
i want to know why profiling with time.time() gives inaccurate results.</p>
<p>an example: <a href="https://github.com/facebookresearch/moco/issues/66" class="inline-onebox" rel="noopener nofollow ugc">Are there results with other normalizations? · Issue #66 · facebookresearch/moco · GitHub</a></p>
<p>Thank you !</p>

username: tom
cooked: <p>It isn’t. While there are more refined measures, there isn’t anything wrong with plain timing. Apparently there first are lots of people doing it wrong (both beginners and people with considerable experience) and then inaccurate representations of what exactly is wrong (“can’t use time.time” <em>Edit:</em> actually it is true you should not use it but time.perf_counter(!)): The main things to get right is warm-up and synchronization.</p>
<p>The thing is that if you use the GPU, unless you call <code>torch.cuda.synchronize()</code> before taking the time (for both start and finish), you don’t know what has been executed before and after the time taking.<br>
I invariably use the following pattern:</p>
<pre><code class="lang-auto">def do_stuff():
   for _ in range(100): # or 1000 or whatever, depending on how long it takes
      do_my_computation()
   torch.cuda.synchronize()

do_stuff()
%timeit do_it()
</code></pre>
<p>Of course, you need to divide the time by whatever size of the loop you have. I usually aim to have something in the msec range or so.</p>
<p>What this does:<br>
This does run the operator (<code>do_my_computation</code>) multiple times between syncs, this would reduce the influence of the synchronization (which takes time) on the measurement.</p>
<p>Calling <code>do_stuff()</code> before the timing does:</p>
<ul>
<li>Warm-up (e.g. some things compile kernels on the fly when called for the first time etc.)</li>
<li>Synchronize before starting the timing</li>
</ul>
<p>Timing with <code>do_stuff()</code> ensures that synchronization happens after each run (and thus implicitly before the next).</p>
<p>You can do essentially the same thing with 
time.time
 time.perf_counter() before and after what is <code>%timeit</code> here, except that timeit will actually call <code>do_stuff</code> several times and do some stats to help you along. There also is the <code>timeit</code> module which is similar but you need to adjust the number of runs manually to the duration of your computation.</p>
<p>That said, the profiler gives you more detailed information with very little effort.</p>
<p>Best regards</p>
<p>Thomas</p>

username: LeeDoYup
cooked: <p>Hello tom. first of all, thank you for fast and detailed reply !</p>
<p>After i read, i tried to use time.time() and torch.cuda.synchronize() to profile the elapsed times.<br>
However, the results are still inaccurate.</p>
<p>For example of <a href="https://github.com/facebookresearch/moco/issues/66" rel="nofollow noopener">https://github.com/facebookresearch/moco/issues/66</a>,</p>
<p>i) <code>time.time()</code> w/o <code>torch.cuda.synchronize()</code></p>
<ul>
<li>shuffle time: 0.5993 s</li>
<li>inf_time: 0.1185 s</li>
</ul>
<p>ii) use <code>torch.cuda.Event</code> &amp; <code>torch.cuda.synchronize()</code></p>
<ul>
<li>shuffle time: 2.72 ms</li>
<li>inf_time: 59.88 ms</li>
</ul>
<p>iii) use <code>time.time()</code> w/ <code>torch.cuda.synchronize()</code></p>
<ul>
<li>shuffle time: 0.0649 s</li>
<li>inf_time: 0.0587 s</li>
</ul>
<p>Although i use torch.cuda.synchronize(),<br>
the shuffle time is still over-estimated.</p>

username: tom
cooked: <p>Ah, wait. The other thing you should do is use <code>time.perf_counter()</code> instead of <code>time.time()</code>. This is because <code>time.time()</code> isn’t guaranteed to actually give valid differences, but you need to use a monotonic clock for that.</p>

username: LeeDoYup
cooked: <p>Thanks Tom.<br>
I checked both <code>time.perf_counter()</code> and <code>time.process_time()</code> with <code>torch.cuda.synchronize()</code>, and got similar results to time.time()</p>
<p>iv) use <code>time.perf_counter()</code> w/ <code>torch.cuda.synchronize()</code></p>
<ul>
<li>shuffle time:	 0.0650 s</li>
<li>inf time:	 0.0587 s</li>
</ul>
<p>v) use <code>time.process_time()</code> w/ <code>torch.cuda.synchronize()</code></p>
<ul>
<li>shuffle time:	 0.0879 s</li>
<li>inf time:	 0.0584 s</li>
</ul>
<p>When comparing all the results, the inference time is consistent,<br>
but the shuffle time is inconsistent by the profiling method.</p>
<p>the shuffle time is shuffleBN is Moco<br>
</p><aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/facebookresearch/moco/blob/78b69cafae80bc74cd1a89ac3fb365dc20d157d3/moco/builder.py#L133" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/facebookresearch/moco/blob/78b69cafae80bc74cd1a89ac3fb365dc20d157d3/moco/builder.py#L133" target="_blank" rel="nofollow noopener">facebookresearch/moco/blob/78b69cafae80bc74cd1a89ac3fb365dc20d157d3/moco/builder.py#L133</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="123" style="counter-reset: li-counter 122 ;">
<li>
</li><li># compute query features</li>
<li>q = self.encoder_q(im_q)  # queries: NxC</li>
<li>q = nn.functional.normalize(q, dim=1)</li>
<li>
</li><li># compute key features</li>
<li>with torch.no_grad():  # no gradient to keys</li>
<li>    self._momentum_update_key_encoder()  # update the key encoder</li>
<li>
</li><li>    # shuffle for making use of BN</li>
<li class="selected">    im_k, idx_unshuffle = self._batch_shuffle_ddp(im_k)</li>
<li>
</li><li>    k = self.encoder_k(im_k)  # keys: NxC</li>
<li>    k = nn.functional.normalize(k, dim=1)</li>
<li>
</li><li>    # undo shuffle</li>
<li>    k = self._batch_unshuffle_ddp(k, idx_unshuffle)</li>
<li>
</li><li># compute logits</li>
<li># Einstein sum is more intuitive</li>
<li># positive logits: Nx1</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
which gathers all samples, shuffles the index, and reallocates mini-batch to each GPU.
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/facebookresearch/moco/blob/78b69cafae80bc74cd1a89ac3fb365dc20d157d3/moco/builder.py#L69-L94" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/facebookresearch/moco/blob/78b69cafae80bc74cd1a89ac3fb365dc20d157d3/moco/builder.py#L69-L94" target="_blank" rel="nofollow noopener">facebookresearch/moco/blob/78b69cafae80bc74cd1a89ac3fb365dc20d157d3/moco/builder.py#L69-L94</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="69" style="counter-reset: li-counter 68 ;">
<li>def _batch_shuffle_ddp(self, x):</li>
<li>    """</li>
<li>    Batch shuffle, for making use of BatchNorm.</li>
<li>    *** Only support DistributedDataParallel (DDP) model. ***</li>
<li>    """</li>
<li>    # gather from all gpus</li>
<li>    batch_size_this = x.shape[0]</li>
<li>    x_gather = concat_all_gather(x)</li>
<li>    batch_size_all = x_gather.shape[0]</li>
<li>
</li><li>    num_gpus = batch_size_all // batch_size_this</li>
<li>
</li><li>    # random shuffle index</li>
<li>    idx_shuffle = torch.randperm(batch_size_all).cuda()</li>
<li>
</li><li>    # broadcast to all gpus</li>
<li>    torch.distributed.broadcast(idx_shuffle, src=0)</li>
<li>
</li><li>    # index for restoring</li>
<li>    idx_unshuffle = torch.argsort(idx_shuffle)</li>
</ol></code></pre>

  This file has been truncated. <a href="https://github.com/facebookresearch/moco/blob/78b69cafae80bc74cd1a89ac3fb365dc20d157d3/moco/builder.py#L69-L94" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>I cannot infer a reason why only the record time of this shuffle operation has varied by measuring method.</p>

username: tom
cooked: <p>That might be a lot clearer if you</p>
<ul>
<li>specify what exactly you want to measure,</li>
<li>time these bits in isolation,</li>
<li>break down the entire thing into the bits you want to measure (i.e. do your parts reconcile to the total? if not, where are overlaps or gaps in the parts).</li>
</ul>
<p>The links don’t seem to show the actual measurement you inserted.</p>

username: LeeDoYup
cooked: <p>In the forward pass of moco, i am measuring the shuffling time, that is</p>
<pre><code class="lang-auto">    def forward(self, im_q, im_k):
        (....)
        # compute key features
        with torch.no_grad():  # no gradient to keys
            self._momentum_update_key_encoder()  # update the key encoder
            
            start_time = time.time()
            # shuffle for making use of BN
            im_k, idx_unshuffle = self._batch_shuffle_ddp(im_k)
            torch.cuda.synchronize()
            end_time = time.time()
            shuffle_time = end_time - start_time
            (...)

</code></pre>

username: tom
cooked: <p>So the immediate takeaway from the above discussion is</p>
<ol>
<li>replace time.time with time.perf_counter()</li>
<li>have a torch.cuda.synchronize() before taking the the start_time,</li>
<li>maybe don’t take the first batch (of a given size)</li>
</ol>

username: LeeDoYup
cooked: <h3>time.time()</h3>
<p>i)  <code>time.time()</code>  w/o  <code>torch.cuda.synchronize()</code></p>
<ul>
<li>shuffle time: 0.5993 s</li>
<li>inf_time: 0.1185 s</li>
</ul>
<p>ii)  <code>time.time()</code>  w/o  <code>torch.cuda.synchronize()</code> both before and after an operation</p>
<ul>
<li>shuffle time: 0.0018 s</li>
<li>inf_time: 0.031 s</li>
</ul>
<h2>time.perf_counter()</h2>
<p>i) use  <code>time.perf_counter()</code>  w/  <code>torch.cuda.synchronize()</code></p>
<ul>
<li>shuffle time: 0.0650 s</li>
<li>inf time: 0.0587 s</li>
</ul>
<p>ii) use  <code>time.perf_counter()</code>  w/  <code>torch.cuda.synchronize()</code> both before and after an operation</p>
<ul>
<li>shuffle time: 0.0021 s</li>
<li>inf time: 0.0309 s</li>
</ul>
<h2>time.process_time()</h2>
<p>i) use  <code>time.process_time()</code>  w/  <code>torch.cuda.synchronize()</code></p>
<ul>
<li>shuffle time: 0.0879 s</li>
<li>inf time: 0.0584 s</li>
</ul>
<p>ii) use  <code>time.process_time()</code>  w/ <code>torch.cuda.synchronize()</code> both before and after an operation</p>
<ul>
<li>shuffle time: 0.001879 s</li>
<li>inf time: 0.03107 s</li>
</ul>
<h2>torch.cuda.Event</h2>
<p>i) use  <code>torch.cuda.Event</code>  &amp;  <code>torch.cuda.synchronize()</code></p>
<ul>
<li>shuffle time: 2.72 ms</li>
<li>inf_time: 59.88 ms</li>
</ul>
<h2>Conclusions</h2>
<ul>
<li>When time profiling, we can use both <code>time</code> module in python and <code>torch.cuda.Event</code>.</li>
<li>What we remember is, we must use <code>torch.cuda.synchronize()</code> right before and after the operations to  be profiled.</li>
<li>Measurements of <code>time</code> and <code>cuda.Event</code> is some what different, but the ratio is consistent.</li>
</ul>

username: LeeDoYup
cooked: <p>In the last,<br>
can i ask some reasons why the <code>time</code> module and cuda.Event measure different elapsed times?<br>
i am using V100x4 and DistributedDataParallel.</p>

username: tom
cooked: <p>I must admit that I don’t know. One thing to exclude would be stochastic variation (e.g. %timeit gives you a standard deviation, so you can imagine error bars for the measurement), but I would not see 30ms-&gt;60ms doing that.<br>
The other part is that you need a really stable environment to get reliable benchmarking, maybe something we did here changed something w.r.t. other things going on.<br>
(But maybe <a class="mention" href="/u/ptrblck">@ptrblck</a> knows something.)</p>
<p>Fun anecdote: A long time ago, I briefly enabled remote access to the GPU I used for benchmarking for one of fellow PyTorch devs because I somehow had a much more stable timing environment then them.</p>
<p>Best regards</p>
<p>Thomas</p>