DataLoader implicitly using CPU?

pband1256: <p>Hi all,</p>
<p>I’ve been noticing severe CPU bottlenecks in my code, where CPU utilization is almost always ~100% while GPU is consistently underutilized. I tried to send all data into GPU in the first place, and the code runs fine but didn’t give any noticeable speed boost, and <code>autograd</code> profiler shows only a 15% decrease in <code>to(device)</code> calls. Afterwards, I noticed that the total size of my data set is significantly larger than available memory on my current card (~34GB of data on a Tesla K80). So I’m wondering if <code>DataLoader</code> is doing some implicit calls moving data from CPU to GPU or vice versa. Another thing that I thought might be the problem is that two elements in <code>collate_fn</code> isn’t moved to GPU (<code>evt_ids, evt_names</code>), so <code>DataLoader</code> has a mix of GPU/CPU elements. Would this make a difference?</p>
<pre><code class="lang-auto">def collate_fn(samples):
    X = [s[0] for s in samples]
    y = [s[1] for s in samples]
    w = [s[2] for s in samples] 
    evt_ids   = [s[3] for s in samples]
    evt_names = [s[4] for s in samples]

    # custom function padding different sized inputs
    X, adj_mask, batch_nb_nodes = pad_batch(X)
    device = torch.device('cuda')

    X = torch.FloatTensor(X).to(device)
    y = torch.FloatTensor(y).to(device)
    w = torch.FloatTensor(w).to(device)
    adj_mask = torch.FloatTensor(adj_mask).to(device)
    batch_nb_nodes = torch.FloatTensor(batch_nb_nodes).to(device)
    return X, y, w, adj_mask, batch_nb_nodes, evt_ids, evt_names

# Loading in multiple files and switch every "epoch"
multi_train_loader = []
for file in args.train_file:
      loader = DataLoader(
                                dataset=file,
                                args.nb_train,
                                batch_size=args.batch_size,
                                shuffle=True,
                                collate_fn=collate_fn,
                                drop_last=True)
      multi_train_loader.append(train_loader)

for i in range(args.nb_epoch):
    train_loader = multi_train_loader[i % len(multi_train_loader)]
    for i, batch in enumerate(train_loader):
        X, y, w, adj_mask, batch_nb_nodes, _, _ = batch

</code></pre>

ptrblck: <aside class="quote no-group" data-username="pband1256" data-post="1" data-topic="86923">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/c4cdca/48.png" class="avatar"> pband1256:</div>
<blockquote>
<p>So I’m wondering if <code>DataLoader</code> is doing some implicit calls moving data from CPU to GPU or vice versa.</p>
</blockquote>
</aside>
<p>No, the <code>DataLoader</code> will load each sample from <code>Dataset.__getitem__</code> and use the <code>collate_fn</code> to create a batch out of these samples. It has no knowledge, if these tensors are on the CPU or GPU.</p>
<p>Your current approach of moving the data to the GPU in your <code>collate_fn</code> should have the same effect as moving the data to the device inside the training loop, since each batch will be pushed to the device, not the complete dataset.</p>
<p>For general information about data loading bottlenecks, I would recommend to have a look at <a href="https://discuss.pytorch.org/t/how-to-prefetch-data-when-processing-with-gpu/548/19">this post</a>.</p>

leiterrl: <p>Hi,</p>
<p>so I would like to revisit this topic as I have run into a similar bottleneck recently.</p>
<p>My setup is a relatively small dataset which I fully push to GPU memory in my Dataset <code>__init__</code> function. Now I too noticed really low GPU utilization when using a standard batched DataLoader.</p>
<p>After some digging and profiling I found that in the current <a href="https://github.com/pytorch/pytorch/blob/6ab1fe19eec0aa610bca6de881d373905da2fb37/torch/utils/data/_utils/fetch.py#L49" rel="noopener nofollow ugc">_MapDatasetFetcher:fetch</a> function the batch is constructed by a list comprehension. Note that <code>self.auto_collation</code> will always be <code>True</code> when passing a <code>batch_size</code> argument to the DataLoader constructor.</p>
<p>This seems to be the root of the bottleneck when looking at a profiling trace where I recreate and compare constructing the batch using either with a list comprehension or just by accessing the whole batch at once from the Dataset instance:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/c/1c906f9ea142e5d7566d91ede379665c3e1d5724.png" data-download-href="https://discuss.pytorch.org/uploads/default/1c906f9ea142e5d7566d91ede379665c3e1d5724" title="profiler_dataloader"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/c/1c906f9ea142e5d7566d91ede379665c3e1d5724_2_690x399.png" alt="profiler_dataloader" data-base62-sha1="44GOXpc6qkJv92QAag1jPTYDsEs" width="690" height="399" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/c/1c906f9ea142e5d7566d91ede379665c3e1d5724_2_690x399.png, https://discuss.pytorch.org/uploads/default/original/3X/1/c/1c906f9ea142e5d7566d91ede379665c3e1d5724.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/c/1c906f9ea142e5d7566d91ede379665c3e1d5724.png 2x" data-dominant-color="E1E0DB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">profiler_dataloader</span><span class="informations">953×552 63.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Here is the corresponding code:</p>
<pre><code class="lang-auto">sampler = RandomSampler(train_dataset)
batch_sampler = BatchSampler(sampler, cfg.params.batch_size, False)

for epoch in range(cfg.params.n_epochs):
    for batch_idx, batch_indices in enumerate(batch_sampler):
        torch.cuda.nvtx.range_push("start fetch style batch retrieval")
        fetch_batch = [train_dataset[idx] for idx in batch_indices]
        fetch_batch_sample = default_collate(fetch_batch)
        torch.cuda.synchronize()
        torch.cuda.nvtx.range_pop()
        
        torch.cuda.nvtx.range_push("start indexed batch retrieval")
        batch_sample = train_dataset[batch_indices]
        torch.cuda.synchronize()
        torch.cuda.nvtx.range_pop()
</code></pre>
<p>So I’m wondering what is the purpouse of that list comprehension in the first place? Looking at a git blame it seems to have always been there even before Iterable-Style Datasets were introduced.</p>