DataLoader behaves differently in the time when initializing an iterator in single-GPU and distributed multi-GPUs configurations

username: T_Qri
cooked: <p>Hi,</p>
<p><strong>Problem</strong></p>
<p>I observe significantly different behaviors in returning an iterator from a multiprocessing dataloader between single-GPU training and multi-GPU distributed training.</p>
<p><strong>Background</strong></p>
<p>I’m using a custom map-style dataset, in which I read a large text file and convert to a list of numpy arrarys and keep them in the memory for later indexing. At the same time, I define a custom batch sampler, which pre-generates a list of batch indices and also keep them in memory, yielding a list of sample indices each time.</p>
<p>Here is how I use the DataLoader:</p>
<pre><code class="lang-auto">    itr=torch.utils.data.DataLoader(memory_dataset, batch_sampler=batch_sampler, collate_fn=Collate(source_vocab.pad_id, target_vocab.pad_id), num_workers=args.num_workers)

    for epoch in range(max_epoch):
        for batch in itr:
            batch=to_cuda(batch)
            ...
</code></pre>
<p>The <code>Collate()</code> is a callable object that gathers a list of numpy arrays from <code>memory_dataset</code> and convert them into CPU tensors.</p>
<p><strong>Profiling</strong>:</p>
<p>I run the above code in 1-GPU and 2-GPU settings and vary the number of workers in dataloader. For 2-GPU setting, I use <code>DistributedDataParallel</code>.</p>
<p>Here I measure the average time in seconds it takes to initialize <code>itr</code> as an iterator object for each epoch in different settings:</p>
<pre><code class="lang-auto">(1) 1-GPUs, 0-workers: 0.7s
(2) 1-GPUs, 1-workers: 0.8s
(3) 1-GPUs, 6-workers: 1.1s
(4) 2-GPUs, 0-workers: 0.8s
(5) 2-GPUs, 1-workers: 37s
(6) 2-GPUs, 6-workers: 210s

</code></pre>
<p><strong>Questions</strong></p>
<ol>
<li>
<p>In PyTorch’s <a href="https://pytorch.org/docs/stable/data.html" rel="nofollow noopener">torch.utils.data documentation</a>, it suggests that for an iterable-style dataset, “<em>When num_workers &gt; 0, each worker process will have a different copy of the dataset object</em>”. I assume it’s the same  for a map-style dataset in my case, so that explains the (4)(5)(6) results, since more workers requires more copy operations in the main process.<br>
But it’not true in 1-GPU settings, why?</p>
</li>
<li>
<p>In PyTorch’s <a href="https://pytorch.org/docs/stable/multiprocessing.html#module-torch.multiprocessing" rel="nofollow noopener">torch.multiprocessing documentation</a>, it mentions that "<em>It registers custom reducers, that use shared memory to provide shared views on the same data in different processes.</em> " If this implies that <code>torch.multiprocessing</code> automatically creates a shared view of memory across sub-processes for arbitrary types, like numpy.ndarray and Python builtin types, I assume by torch.multiprocessing, each worker of the DataLoader does not hold a different copy of the dataset, but accesses the dataset through a shared memory. So, that explains the (1)(2)(3) results, since no copy is required, the time to initialize different number of workers has nothing to do with the size of the dataset.<br>
But (4)(5)(6) grows almost linearly with the number of workers, why is that?</p>
</li>
<li>
<p>Overall, I couldn’t figure out why DataLoader behaves so much differently in single-GPU and distributed multi-GPUs configurations.</p>
</li>
</ol>
<p>Any suggestions would be appreciated!</p>

username: ptrblck
cooked: <p>Is <code>memro_dataset</code> loading the data in its <code>__init__</code> or does it already contain all the data?<br>
Each worker would create a copy of this dataset, which would explain the slight increase in 1, 2, 3.</p>
<p>However, each process in the DDP setup would create the specified number of workers, so that you would end up with more workers in the end. I don’t know, why the creation takes so much more time though.</p>

username: T_Qri
cooked: <p>Thanks for your attention.  <code>memory_dataset</code>  loads all the data in its  <code>__init__</code>  function. And I also expected the same performance in either single-GPU case or DDP multi-GPU case.</p>