Guidelines for assigning num_workers to DataLoader

abweiss: <p>I realize that to some extent this comes down to experimentation, but are there any general guidelines on how to choose the num_workers for a DataLoader object?  Should num_workers be equal to the batch size?   Or the number of CPU cores in my machine?  Or to the number of GPUs in my data-parallelized model?  Is there a tradeoff with using more workers due to overhead?  Also, is there ever a reason to leave num_workers as 0 instead of setting it at least to 1?</p>

apaszke: <p>Having more workers will increase the memory usage and that’s the most serious overhead. I’d just experiment and launch approximately as many as are needed to saturate the training. It depends on the batch size, but I wouldn’t set it to the same number - each worker loads a single batch and returns it only once it’s ready.</p>
<p>num_workers equal 0 means that it’s the main process that will do the data loading when needed, num_workers equal 1 is the same as any n, but you’ll only have a single worker, so it might be slow</p>

mjchen611: <p>I use multi subprocesses to load data(num_workers =8) and with the increase of epoch,I notice that the (RAM, but not GPU) memory increases.<br>
I thought may be I can kill subprocesses after a few of epochs and then reset new subprocesses to continue train the network,but I don’t know how to kill the subprocesses in the main processes.</p>
<p>And I set num_workers = 0,the (RAM, but not GPU) memory remains stable with the increase of epoch.</p>
<p>Can you give me some suggestions or instructions about the problem?</p>
<p>Thank you so much.</p>

YossiB: <p>Are you sure that memory usage is the most serious overhead ? What about IO usage ?<br>
Setting too many workers might cause seriously high IO usage which can become very uneffective.</p>
<p>I would love to get your advice about the recommended way to deal with my data - I feed my CNN with large batches (256/512/1024…) of small patches of size 50x50. I intend to use the ImageFolder DataLoader for that, but I’m afraid that it would be very uneffective to load from disk a lot of small images in high frequency.</p>

harsv: <p>I experimented with this a bit. I found that we should use the formula:<br>
num_worker = 4 * num_GPU .<br>
Though a factor of 2 and 8 also work good but lower factor (&lt;2) significantly reduces overall performance. Here, worker has no impact on GPU memory allocation. Also, nowadays there are many CPU cores in a machine with few GPUs (&lt;8), so the above formula is practical.</p>

taiky: <p>Is it right to estimate this from data throughput?<br>
data load by CPU per batch == data process by GPU per batch<br>
=&gt;<br>
entry_KB * batch_size * num_worker = num_GPU * GPU_throughput</p>

Brando_Miranda: <p>if the data set is small like cifar10, why doesn’t the whole data set stay in the GPU the whole time? Why would # workers do anything?</p>

ptrblck: <p>The more data you put into the GPU memory, the less memory is available for the model.<br>
If your model and data is small, it shouldn’t be a problem. Otherwise I would rather use the <code>DataLoader</code> to load and push the samples onto the GPU than to make my model smaller.</p>

Eric_K: <p>hello,<br>
For example, if one worker loads a single batch expends 1.5s and one iteration in GPU expends 0.5s. Are there 3 workers optimal in your opinion?<br>
Thanks~</p>

Brando_Miranda: <p>Are you saying that if the data and model are both <em>small</em> the dataloader class isn’t the right thing to use?</p>

ptrblck: <p>If your dataset is really small and you don’t need batching, you can just push the data onto the GPU and simply apply your training procedure.</p>
<p>However, since I like the concept of a <code>Dataset</code> and <code>DataLoder</code>, I would still use a <code>DataLoader</code> in such a use case just to be able to easily extend the dataset and use batching, shuffling etc.</p>

Brando_Miranda: <p>you could check how many cpus and cores u have with <code>lscpu</code> if u want an initial guess without doing benchmarking…</p>

singleroc: <p>From <a href="https://pytorch.org/docs/master/data.html" rel="nofollow noopener">https://pytorch.org/docs/master/data.html</a><br>
it could be known that:<br>
number_worker is the subprocess count. If memory_pin not true, it only increase the CPU DDR memory rather the GPU memory. If memory_pin is true, the GPU memory would increase also.</p>
<p>Correct me if you have a different opinion.</p>

ptrblck: <p>I’m not sure about the increase in GPU memory.<br>
As I understand, pinned memory is used as a staging area on the host side (CPU).<br>
So if <code>pin_memory=True</code>, the data will be directly copied to the pinned memory and from there to the GPU.<br>
If <code>pin_memory=False</code>, the data will be allocated in pageable memory, transferred to the pinned memory, and then to the GPU.<br>
See the <a href="https://devblogs.nvidia.com/how-optimize-data-transfers-cuda-cc/">NVIDIA devblog on pinned memory</a>.</p>

mnazaal: <p>Just wanted to mention something I noticed;<br>
setting num_workers=1 gave me a “cuda runtime error (2) out of memory” exception, and increasing it helped.</p>
<p>Cheers</p>

isalirezag: <p>Also for unknown reason i notic increasing the num_workers give me nan in my loss <img src="https://discuss.pytorch.org/images/emoji/apple/man_shrugging/2.png?v=5" title=":man_shrugging:t2:" class="emoji" alt=":man_shrugging:t2:"></p>

niez01: <p>Is there any one has met this situation that setting num_workers = 4 could make the train stop? Recently, I tested a RFBnet project, and find when I set num_workers= 4 will stop training at epoch = 2. However, num_workers=0 will be fine.</p>

devforfu: <p>Not sure what is the reason but I am quite often getting <code>MemoryError</code> exception when using <code>num_workers != 0</code>. Could somebody describe how this process usually works? Does it copy dataset instance (including all its properties) into subprocess? Or does it use threads?</p>
<p>It seems that during the training process the amount of free RAM continues to reduce. I am using a custom dataset that generates images from strokes (Quick Draw Doodles data), and probably the problem is that the dataset doesn’t work well in multitasking setting. Could somebody give an advice on how to implement a multithread ready dataset?</p>

kuzand: <p>What’s num_GPU? How to get it on google colab?</p>

Junhao_Wen: <p>Hi,</p>
<p>I encountered the similar problem for DataLoader.</p>
<p>Sorry to ask the similar question, after reading all your discussion, I am still confused about the relationship between the number_GPU, num_CPU and the num_works.</p>
<p>Here is my situation:</p>
<p>I have a convolutional Autoencoder (so it’s not a very big network), but I have a very big dataset: For one epoch, I have `</p>
<blockquote>
<p>16 (batch_size) * 7993 = 12788</p>
</blockquote>
<p>images, each image’s dimension is <code>51 x 51 x 51</code>.</p>
<p>So I used one GPU (Tesla P100) and set the <strong>num_workers=8</strong>. I also tried other options for num_works, like 0 or 16.</p>
<p>Always, it is very slow to load the data, the training time for each batch is very fast. To load each image, I found that with num_workers=8, it will take longer than num_workers=0. Every time, I will take a long time to load 74 batches and run the training, then reload another 74 batches (the case of num_workers=8).</p>
<p>Do you have any suggestions to set the num_workers if the dataset is huge???</p>
<p>Thanks in advance</p>