Time profiling for input feeding to model?

samra-irshad: <p>Hi, I am trying to assess how much time my code takes to acquire the input (with data augmentation). Currently I am using Pytorch’s dataloader. I started recording the time immediately after <code>__getitem__</code> method till it returned a batch. Is it the right way to assess the time?<br>
My code looks something like this:</p>
<pre><code class="lang-auto">def __getitem__(self, idx):
    ts = time.time()
......
......
print("time elapsed for inputs{}s".format(time.time() - ts))
   return batch
</code></pre>

ptrblck: <p>Your approach could work to profile the loading of a single sample.<br>
However, if you want to profile the data loading via a <code>DataLoader</code> (and thus potentially multiple workers), I would recommend to use <a href="https://github.com/pytorch/examples/blob/2c57b0011a096aef83da3b5265a14db2f80cb124/imagenet/main.py#L279-L282">this code snippet</a> from the <code>ImageNet</code> example.</p>

samra-irshad: <p><em>single sample</em> or a <em>single batch</em>?</p>

ptrblck: <p>The <code>__getitem__</code> method is usually used to load a single sample using the passed <code>index</code>.<br>
So your code snippet would profile the loading time of a single sample, while the ImageNet example would profile the complete <code>DataLoader</code>, i.e. the time could approach zero if the workers are preloading the next batches fast enough while your GPU is busy.</p>

samra-irshad: <p>Right. So improving the loading speed of a single example correlates to speeding up the whole batch speed and subsequently training? I tried that ImageNet example, but I am confused between <em>data time</em> and <em>batch time</em>. So according to example:</p>
<blockquote>
<p>ts = time.time()<br>
for iter, batch in enumerate(train_loader):<br>
data_time.update(time.time() - ts)</p>
<p>…<br>
…<br>
outputs = fcn_model(inputs)<br>
labels = labels.type_as(outputs)<br>
loss = criterion(outputs, labels)<br>
loss.backward()<br>
optimizer.step()<br>
batch_time.update(time.time() - ts)</p>
</blockquote>
<p>So data time is the time taken by dataloader to load a batch? and batch time is time taken by model to process a batch? I think I can only improve the efficiency of data loading, not batch processing (batch time) because that depends on model, right?</p>

ptrblck: <p>Yes, your statement is correct.<br>
The first iteration of the <code>DataLoader</code> would be slower since all workers are loading a complete batch. If the data loading is not a bottleneck, the data loading time should decrease towards zero.</p>
<p>To accelerate the model, you could use e.g. mixed-precision training and check if this would yield a speedup.<br>
Also, <code>torch.backends.cudnn.benchmark = True</code> would enable cudnn to profile the kernels for each new input shape and could accelerate the training.</p>

zaccharieramzi: <p>Regarding this code snippet that you highlighted: isn’t there a potential problem with this way of measuring the data load time because of <code>non_blocking=True</code>?</p>
<p>Basically, this won’t take into account the CPU to GPU transfer, no?</p>

ptrblck: <p>The linked timer is not used to explicitly measure the time it takes to load and process each batch and to send it to the device. Instead it measures how much time of your training loop is spent on waiting on the next batch. If your use case is not suffering from a data loading bottleneck this timer would decrease towards zero in the optimal case, since the next batch would already be preloaded and you wouldn’t have to wait for it.<br>
In this sense you are correct that we are ignoring the host to device copy, but we also don’t want to measure it.</p>