Dataloader is more than 4x slower

username: mkserge
cooked: <p>Hi,</p>
<p>I have a codebase that is using a custom Iterator over a numpy array. The iterator extends <code>torch.utils.data.IterableDataset</code> and implements <code>__next__</code> and <code>__iter__</code> methods of the super class.</p>
<p>I am trying to switch to <code>Dataset</code> + <code>Dataloader</code> implementation of the same setup. I bring a simplified example below.</p>
<pre><code class="lang-auto">class NumpyArrayDataset(Dataset):
    def __init__(self) -&gt; None:
        self.data = np.random.randint(low=[0, 0, 0], high=[1226, 1226, 29517], size=(66038, 3))
    def __len__(self):
        return len(self.data)
    def __getitem__(self, idx):
        return self.data[idx]
</code></pre>
<p>And the use this in my code with</p>
<pre><code class="lang-auto">    dataset = NumpyArrayDataset()
    dataloader = DataLoader(
        dataset=dataset,
        batch_size=batch_size,
        shuffle=True
    )
</code></pre>
<p>In the iterable dataset scenario each batch is pushed onto the device by constructing a tensor from numpy array directly on the GPU. While in the Dataset scenario, since PyTorch automatically returns Tensors if the data is numpy array I simply push the data onto the device with <code>.to(device)</code> after retrieving each batch.</p>
<p>This switch alone causes more than 3x drop in the speed of training, going from around <code>632,214 samples/second</code> to <code>182,106 samples/second</code> speeds.</p>
<p>The training job is running in Data Parallel setup on a node with 8 V100 GPUs. Admittedly, GPU utilization is pretty low (2% on the main and around 1% on the rest).</p>
<p>I tried experimenting with <code>pin_memory=True</code> with various <code>num_workers</code> results in much, much worse speed degradations.</p>
<p>The goal is to migrate to DDP, and DDP requires Dataloader with a Sampler, so hence my efforts to switch.</p>
<p>Any ideas on what might be the cause of this?</p>

username: J_Johnson
cooked: <p>Dataset and Dataloader classes are most appropriate with data that is accessed from your hard drive. But there are issues in Windows operating systems when setting <code>num_workers</code> ≠ 0.</p>
<p>If you can upload all of your data into ram or, better yet, onto a gpu, you can use something like this:</p>
<aside class="quote quote-modified" data-post="2" data-topic="165070">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/j_johnson/48/55494_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/use-case-for-loading-the-entire-dataset-into-ram/165070/2">Use case for loading the entire dataset into RAM</a> <a class="badge-category__wrapper " href="/c/data/37"><span data-category-id="37" data-drop-close="true" class="badge-category " title="Topics related to DataLoader, Dataset, torch.utils.data, pytorch/data, and TorchArrow."><span class="badge-category__name">data</span></span></a>
  </div>
  <blockquote>
    Just a suggestion: don’t use the vanilla dataloader for this. If you can fit the data all into ram, just turn it into a tensor for each set of data, shape being: 
self.dataset.size()=([data_index, ...])

Where each data_index is a sample. 
Then create an indexer, either in NumPy or Pytorch. Here is an example for NumPy. 
rng=np.random.default_rng()

...
self.data_index=np.arange(self.dataset.size()[0])

Then define a shuffler class you can call whenever you want to shuffle the index: 
def shuffl…
  </blockquote>
</aside>


username: mkserge
cooked: <p>Hi,</p>
<p>In both scenarios I have the entire dataset loaded into memory. The code is running on a Linux machine. In a training loop, I push every batch onto a GPU. The only difference is that with the custom iterator I build a tensor directly on a GPU, since the iterator returns a slice of a numpy array</p>
<pre><code class="lang-auto">    source = torch.tensor(data[:, 0], dtype=torch.int64, device=device, requires_grad=False)
    target = torch.tensor(data[:, 1], dtype=torch.int64, device=device, requires_grad=False)
</code></pre>
<p>While with dataset/dataloader setup, I only push it onto a GPU, since PyTorch automatically converts the numpy into a Tensor.</p>
<pre><code class="lang-auto">    source = data[:, 0].to(device)
    target = data[:, 1].to(device)
</code></pre>
<p>I am just curious what is it in the Dataloader that is causing such a crazy slowdown. I tried running the model on a single GPU (no DataParallel) and the difference is even more striking. On a single GPU, the iterator runs at <code>2,784,729 samples/second</code>, while the Dataset/Dataloader approach runs <code>206,568 samples/second</code>. Pinning memory or changing number of workers only causes slow-down.</p>
<p>Edit: So batching is what appears to be the culprit. In the iterator implementation the <code>next()</code> returns the batch, while in the Dataloader approach batching happens “behind the scenes”. I tried dropping the batch sizes to 1 in both scenarios and speeds are somewhat comparable (Dataloader is still a tad slow at <code>4,663,136 samples/second</code> while the iterator is at <code>4,861,653 samples/second</code>).</p>
<p>Any ideas on what could be causing such a slow down when batching in the Dataloader?</p>
<p>Alternatively can I use a custom iterator that returns a batch in the <code>next()</code> method with DDP? How would i go implementing the Sampler?</p>

username: J_Johnson
cooked: <p>The main advantage to the DataLoader is loading files from a hard drive. It does so iteratively because each process can be assigned to cpu cores with the num_workers argument.</p>
<p>If I’m not mistaken, the next() call will just return 1 sample from the vanilla data loader.</p>
<p>Regarding DDP, in the example given <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" rel="noopener nofollow ugc">here</a>, there is no need to split data for each GPU. You just pass in the train data into the model and Pytorch will handle distribution behind the scenes.</p>
<p>On a side note, you may get an additional speedup if you can keep your custom slicing process handled on a GPU as Pytorch tensors. Pytorch tensors on GPU can be sliced by NumPy arrays from CPU, so you can also take advantage of NumPy’s shuffling function on the index. This can be especially beneficial if your dataloading pipeline applies any calculation heavy augmentations or transforms.</p>

username: mkserge
cooked: <aside class="quote no-group">
<blockquote>
<p>The main advantage to the DataLoader is loading files from a hard drive. It does so iteratively because each process can be assigned to cpu cores with the num_workers argument.</p>
</blockquote>
</aside>
<p>I see, thanks! That explains why num_workers does not help in my scenario since all the data is already in memory.</p>
<aside class="quote no-group">
<blockquote>
<p>Regarding DDP, in the example given <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" rel="noopener nofollow ugc">here</a>, there is no need to split data for each GPU. You just pass in the train data into the model and Pytorch will handle distribution behind the scenes.</p>
</blockquote>
</aside>
<p>From <a href="https://discuss.pytorch.org/t/ddp-without-distributed-sampler/114143">this post</a> it looks like it will not shard the data and iterate over the entire dataset if you don’t pass a DistributedSampler <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>