Issue with dataloader using pin_memory = True

username: ankur6ue
cooked: <p>Hello, I’m seeing an odd issue with using the pin_memory = true flag with the dataloader. I’m measuring the time taken to transfer data from the host RAM to GPU memory as follows:</p>
<pre><code>    transfer_time_start = time.time()
    input = input.cuda(args.gpu, non_blocking=False)
    target = target.cuda(args.gpu, non_blocking=False)
    torch.cuda.synchronize()
    transfer_time.update(time.time()-transfer_time_start)
</code></pre>
<p>with pin_memory = True in the dataloader, this gives me a transfer time of 0.03 sec, which for a batch size of 256, translates into 256<em>224</em>224<em>3</em>4/0.03 = 5.1GB, which is a bit low for my CPU-GPU interconnect (x16, PCIe3) which should deliver ~12GB.</p>
<p>I then tried calling pin_memory() manually on the tensor returned by the enumerate call, as shown below:</p>
<p>for i, (input, target) in enumerate(train_loader):<br>
input = input.pin_memory()<br>
# measure data loading time<br>
data_time.update(time.time() - end)<br>
transfer_time_start = time.time()</p>
<pre><code>    input = input.cuda(args.gpu, non_blocking=False)
    target = target.cuda(args.gpu, non_blocking=False)
    torch.cuda.synchronize()
    transfer_time.update(time.time()-transfer_time_start)
</code></pre>
<p>Now the transfer time dropped to 0.014, which translates to ~11GB, which is as expected. Anyone has any ideas why setting pin_memory = True in the data loader may not return a tensor already in pinned memory?</p>
<p>Also attached below are two plots showing the transfer time (green plot) from host memory to the GPU.<br>
This plot shows the transfer time when I call pin_memory manually<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/1/1030a1bf0654ce5ea04d5ffb376e2196181a852c.png" data-download-href="https://discuss.pytorch.org/uploads/default/1030a1bf0654ce5ea04d5ffb376e2196181a852c" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/1/1030a1bf0654ce5ea04d5ffb376e2196181a852c_2_690x208.png" alt="image" data-base62-sha1="2jdPCZgJNcjw5ZK4YqflrDTq6gQ" width="690" height="208" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/1/1030a1bf0654ce5ea04d5ffb376e2196181a852c_2_690x208.png, https://discuss.pytorch.org/uploads/default/optimized/2X/1/1030a1bf0654ce5ea04d5ffb376e2196181a852c_2_1035x312.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/1/1030a1bf0654ce5ea04d5ffb376e2196181a852c_2_1380x416.png 2x" data-dominant-color="F8F6F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1585×478 111 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
You can see that the transfer time stays consistently low.</p>
<p>Whereas this one shows the transfer time without calling pin_memory manually. Now the transfer time is highly variable and averages to around 0.03 sec<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/0/0637d60222fd949df0229830bbdf1c86eaa467f3.png" alt="image" data-base62-sha1="T0uysPWY4omqgGG0H8V3gYyMGn" width="624" height="247"></p>

username: rasbt
cooked: <p>I can not speak much about the manual approach as I haven’t tried it, but regarding <code>pin_memory=True</code> I observe in practice that it slows done the training at about 2x (compared to <code>False</code>) – tested it in PyTorch 0.4.1 and 1.0 and on two independent machines (one with 1080Ti’s and one with Titan V’s). So, in practice, I abandoned using that. I remember there was a thread where someone mentioned similar observations.</p>
<p>So, it may well be that there’s a bug with <code>pin_memory = True</code>, esp. since you observe that the manual approach results in the expected speed up.</p>

username: ankur6ue
cooked: <p>Thanks for the reply. As my experiments confirm, transfer to GPU is significantly faster for data in pinned memory, so it is worth doing it. Issue is that transfer to pinned memory itself costs time, and only saves time overall if it can be parallelized. The data loader seems to be doing this - it spins up a separate thread for transfer to pinned memory when pin_memory flag is set to True. When you call enumerate or next(iter), dataloader waits until a batch is available in pinned_memory so if the processing time on GPU is sufficiently long, then the latency of pinned memory transfer should be hidden, at least partly.</p>
<p>The question is why is transfer to the GPU still slow even though the batch is in pinned memory? One difference between the manual approach and the regular approach (not calling input.pin_memory() manually) is that in the manual approach, the transfer is done over the main thread, while in the regular approach, it is being done on another thread. Does this make a difference?</p>

username: kirk86
cooked: <p>I can also verify this since I have the same observations were using <code>pin_memory=True</code> and <code>num_workers=1</code> I see the gpu utilization at ~40% through all the training period but with <code>pin_memory=False</code> and <code>num_workers=4</code> the gpu utilization is at ~90%. Plus I see my cpu at full utilization since the fans kick in but without pin_memory everything seems fine.</p>