Does DDP send full dataset to each GPU?

dkoterwa: <p>Hi,<br>
I think that I have a really simple question for most of you. I <strong>am trying to create a Masked Language Modeling script with DDP,</strong> it means that I am tokenizing and masking large portion of text.<br>
The problem I have is that I do not understand one aspect of DDP. I thought that only one process stores full data and then it distributes mini batches to each GPU.</p>
<p><strong>For 3000 examples, when I do:</strong><br>
<code>len(train_dataloader) * num_gpus * batch_size</code> I get 2100 (70% of full dataset is training data). I understand this fragment of text. When I spawn the process on each GPU, the dataloaders store mini batches of data.</p>
<p><strong>However, when I execute:</strong><br>
<code>len(train_dataloader.dataset) + len(eval_dataloader.dataset)</code><br>
Then I receive 3000, which is the size of my full dataset.</p>
<p>How does it exactly work? Each GPU receives full dataset, and then <code>DistributedSampler</code> distributes everything in the <code>DataLoader</code> (thanks to that each <code>DataLoader</code> passes only specific portion of data) but no matter what <code>DataLoader.dataset</code> will contain full data? Can someone explain this aspect more clearly to me? <strong>Does it mean that DDP is not that memory-efficient, but only enables faster training thanks to gradient averaging across multiple GPUs?</strong></p>

ptrblck: <p><code>DDP</code> uses a single process per device in the recommended setup, which means that each process is responsible for loading and processing its own dataset.<br>
You would usually use a <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.distributed.DistributedSampler"><code>DistributedSampler</code></a> which splits the indices based on the number of ranks to avoid using the same data.</p>

dkoterwa: <p>Thank you for your answer. I am using <code>DistributedSampler,</code> and that’s why each <code>DataLoader</code> passes its own data (even when it stores the full dataset when checking <code>dataset</code> argument.<br>
However, I am using spawn method and initializing training for couple of processes. <strong>I also have functions that are loading the dataset inside of the training script. Does it mean that it allocates the full dataset on every GPU?</strong></p>

ptrblck: <p>If you are eagerly preloading the data, each process would also hold it, yes. The common approach is thus to lazily load the data. The data won’t be automatically moved to the GPU and will reside where originally loaded, so the CPU in the common approach.</p>

dkoterwa: <p>Thank you. Could you explain what does lazy loading exactly mean? I cannot find exact information about using it in the Deep Learning field.</p>

ptrblck: <p><a href="https://pytorch.org/tutorials/beginner/basics/data_tutorial.html#creating-a-custom-dataset-for-your-files">This tutorial</a> explains how to write a custom <code>Dataset</code> which lazily loads the data.<br>
Basically you are not preloading all samples during the creation of the <code>Dataset</code>, but are only loading the paths and assign the transformations in the <code>__init__</code> method. While iterating the <code>Dataset</code> directly or via a <code>DataLoader</code> the <code>Dataset.__getitem__</code> method is used to load and process each sample, which is referred to as lazy loading.</p>