When to set pin_memory to true?

kindlychung: <p>From the imagenet <a href="https://github.com/pytorch/examples/blob/master/imagenet/main.py#L133-L145" rel="nofollow noopener">example</a>:</p>
<pre><code class="lang-auto">    train_loader = torch.utils.data.DataLoader(
        train_dataset, batch_size=args.batch_size, shuffle=(train_sampler is None),
        num_workers=args.workers, pin_memory=True, sampler=train_sampler)

    val_loader = torch.utils.data.DataLoader(
        datasets.ImageFolder(valdir, transforms.Compose([
            transforms.Resize(256),
            transforms.CenterCrop(224),
            transforms.ToTensor(),
            normalize,
        ])),
        batch_size=args.batch_size, shuffle=False,
        num_workers=args.workers, pin_memory=True)
</code></pre>
<p>Both data loaders set <code>pin_memory</code> to true, while the default value is false. Are there good reasons to do this?</p>

ptrblck: <p>If you load your samples in the <code>Dataset</code> on CPU and would like to push it during training to the GPU, you can speed up the host to device transfer by enabling <code>pin_memory</code>.<br>
This lets your <code>DataLoader</code> allocate the samples in page-locked memory, which speeds-up the transfer.<br>
You can find more information on the <a href="https://devblogs.nvidia.com/how-optimize-data-transfers-cuda-cc/">NVIDIA blog</a>.</p>

kindlychung: <p>Thanks. This means <code>pin_memory</code> should always be true if you are training on a Nvidia GPU, right?</p>

ptrblck: <p>Generally, yes, if you are loading your data on the CPU.<br>
It won’t work if your dataset is really small and you push it already to the GPU in the <code>Dataset</code>, but that’s pretty obvious.</p>

cakeeatingpolarbear: <p>If I store the paths to my images in my dataset (as I can’t fit my dataset into RAM) and only load the images in the <strong>getitem</strong> function is it still beneficial to use pin memory?</p>

ptrblck: <p>Yes. As far as I know, that’s the exact use case for speeding up the transfer between the host and device.</p>

Ravi_Tej_Akella: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Surprisingly, I am getting better speed-ups with pin_memory=False. Also, the CPU memory usage has also reduced by 2 or 3 folds with pin_memory=False. Just curious what might be the reason.</p>

ptrblck: <p>We had a recent <a href="https://github.com/pytorch/pytorch/issues/25010" rel="nofollow noopener">issue</a>, which should have been fixed by now. Are you seeing the same behavior using the latest nightly build?</p>

Ravi_Tej_Akella: <p>This issue occurs in Pytorch 1.1.0 version. I have not tried out the latest nightly version. My device has CUDA 9.0 (I cannot upgrade CUDA), which I guess is not supported by the latest nightly version.</p>

ptrblck: <p>The binaries ship with their own CUDA and cudnn versions so that you don’t have to install CUDA locally (just the NVIDIA drivers).</p>

Ravi_Tej_Akella: <p>Oh okay. Thanks for the help <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

PySimon: <p>Hi,</p>
<p>when setting pin_memory=True, many more processes are forked than with pin_memory=False, even with num_workers=0 in all cases. What is the reason for that? For me pin_memory=False is much faster in this case.</p>
<p>The tensors come out of my dataloader as CPU tensors and before running my network I call to(device=MY_GPU_ID). Is that wrong? How else should I specify on which GPU I want the tensor to be?</p>
<p>Pytorch is 1.2.0</p>
<p>Best,<br>
Simon</p>

ptrblck: <p>We had an issue using too many threads when <code>pin_memory=True</code> was set recently, which should be fixed in the latest release.<br>
Could you update your PyTorch installation and check the CPU usage again?</p>

Shubhankar: <p>On PyTorch 1.13 I am still getting faster runtimes with <code>pinned_memory=False</code> and using num_workers=20. I have 4 NVIDIA P100 with 256 GB IBM Power8.</p>

AndreaCatania: <p>The tensors that comeout from the <code>DataLoader</code> are on CPU, (at least this is what <code>print(t.device)</code> says), even if:</p>
<pre><code class="lang-auto">num_workers=1,
pin_memory=True
</code></pre>
<p>What’s the correct way to upload these directly on the GPU?</p>

ptrblck: <p><code>pin_memory=True</code> uses pinned host memory for faster copies as described <a href="https://devblogs.nvidia.com/how-optimize-data-transfers-cuda-cc/">here</a>.<br>
Push your tensors via <code>x =x.to('cuda')</code> to the device.</p>

devforfu: <p>Sorry for bumping up this rather old thread. But how does this work with two and more GPUs?</p>
<p>In my use case, I have two experiments that run in parallel, two different models. I use <code>pin_memory=True</code> in one of my experiments with <code>cuda:0</code> device. Then I run one more experiment but with <code>cuda:1</code>, and it fails with OOM error because of making attempt to allocate the memory on another GPU. Is it an expected behavior?</p>

raharth: <p>Hi! I stumbled upon this thread, today wondering about the slowing down of my training.It looks as if the same problem still exists. When setting pin_memory=True it processes roughly half as many iterations as if set to False.</p>
<p>I’m using CUDA10.2 and PyTorch 1.6.0 on Win10 on a K80. My dataset is fairly small with just around 300MB, could that be the reason?</p>

Emre_Yalcinoglu: <p>Thanks for the answers. I wanted to clarify one more thing:<br>
I have a setup where I push the model to the GPU, and then push the input tensors to the GPU <strong>only</strong> during the training like so:</p>
<pre><code class="lang-auto">def train(model, dataloader):
    for epoch in range(num_of_epochs):
        for image_batch, label_batch in dataloader:
            image_batch, label_batch = image_batch.to(device), label_batch.to(device)
            optimizer.zero_grad()
            ...
</code></pre>
<p>However, I am not observing any speed-up. Not even after the first epoch, which I thought how <code>pin_memory=True</code> would affect the process.<br>
So I have two further questions:</p>
<ol>
<li>Is pushing to the device during <code>__getitem__</code> the best-practice? In SO, I saw a comment stating otherwise. If not, then was my example, pushing during training loop, correct?</li>
<li>Do we have to state <code>num_workers</code> in order to get performance help from <code>pin_memory=True</code>?</li>
</ol>

ptrblck: <ol>
<li>
<p>If you want (and/or have to) to lazily load the data, then I would stick with the <code>Dataset</code> approach and load each sample in the <code>__getitem__</code>. You could e.g. preload the complete MNIST dataset and might even be able to push it to the GPU before the training begins (which would give you a performance benefit during training), but that’s of course not always possible especially if you are working with larger datastes.</p>
</li>
<li>
<p>Try to use <code>non_blocking=True</code> in the <code>to()</code> operation.</p>
</li>
</ol>