Negative iteration time measurements

lbhm: <p>I am currently doing some experiments where I want to measure data loading times and compute times (i.e. forward/backward pass) for a couple of models (starting of with Resnet50 on ImageNet). However, I sometimes encounter <em>negative</em> time measurements in my experiments and would like to understand if this issue is related to PyTorch or rather core python components or something else in my code base.</p>
<p>My code is based on the <a href="https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Classification/ConvNets" rel="noopener nofollow ugc">deep learning examples from NVIDIA</a> which in turn are based on an example from the PyTorch community. The core of my time measurements looks like this:</p>
<pre><code class="lang-auto">    for i, (data, label) in data_iter:
        batch_size = data.size(0)
        lr_policy(optimizer, i, epoch)
&gt;&gt;&gt;     data_time = time.time() - end

        optimizer_step = ((i + 1) % batch_size_multiplier) == 0
        # Forward/backward happens in step()
        loss = step(data, label, optimizer_step=optimizer_step)

&gt;&gt;&gt;     iteration_time = time.time() - end

        if logger is not None:
            &lt;some logging stuff&gt;

&gt;&gt;&gt;     end = time.time()
</code></pre>
<p>My logs then show measurements like this:</p>
<pre><code class="lang-auto">Train [1/1492] t.data_time : 0.00018 s t.compute_time : 0.26296 s
Train [1/1493] t.data_time : 0.00018 s t.compute_time : 0.26300 s
Train [1/1494] t.data_time : 0.00018 s t.compute_time : 0.26295 s
Train [1/1495] t.data_time : 0.00018 s t.compute_time : 0.26285 s
Train [1/1496] t.data_time : 0.00018 s t.compute_time : -2.66764 s
Train [1/1497] t.data_time : 0.00018 s t.compute_time : 0.26303 s
Train [1/1498] t.data_time : 0.00017 s t.compute_time : 0.26280 s
Train [1/1499] t.data_time : 0.00017 s t.compute_time : 0.26296 s
</code></pre>
<p>I would greatly appreciate some insight of where these negative measurements could come from. If it helps, I can also provide more information on my experiments and code base.</p>

ptrblck: <p>Is <code>t.compute_time</code> just printing <code>iteration_time</code>?<br>
Also, are you using the <code>logger</code> or did you add the timing code manually to the script?</p>

lbhm: <p><code>t.compute_time</code> is printing the difference between <code>iteration_time</code> and <code>data_time</code>.</p>
<p>I’m using the <code>logger</code> from the NVIDIA script that I mentioned above with <a href="https://github.com/NVIDIA/dllogger" rel="noopener nofollow ugc">DLLogger</a> as the logging backend. The logging code that I excluded in my initial post is:</p>
<pre><code class="lang-auto">        if logger is not None:
            logger.log_metric("t.loss", to_python_float(loss), batch_size)
            logger.log_metric("t.data_speed", calc_speed(batch_size, data_time))
            logger.log_metric("t.compute_speed", calc_speed(batch_size, iteration_time - data_time))
            logger.log_metric("t.iteration_speed", calc_speed(batch_size, iteration_time))
            logger.log_metric("t.data_time", data_time)
            logger.log_metric("t.compute_time", iteration_time - data_time)
            logger.log_metric("t.iteration_time", iteration_time)
</code></pre>
<p>The print logs are actually much larger but above I only showed the relevant part for conciseness.</p>

ptrblck: <p>Thanks for the update. I don’t see any definition of <code>calc_speed</code> in the repository, so could you link to its implementation?</p>

lbhm: <p>I refactored some of the names in my code to be consistent with the rest of my project. <code>calc_speed</code> in my snippet equals <a href="https://github.com/NVIDIA/DeepLearningExamples/blob/master/PyTorch/Classification/ConvNets/image_classification/training.py#L483-L489" rel="noopener nofollow ugc"><code>calc_ips</code></a> in the original code.</p>

ptrblck: <p>Thanks for the update. In that case I don’t know what might cause the wrongly reported time, as the input values shouldn’t get negative (batch size as well as world size) and even if you are not synchronizing the code properly to measure the GPU time, you should not get negative results.<br>
Maybe I’m missing something and someone else has an idea.</p>