Reuse of DataLoader worker process and caching in DataLoader

wentian_zhao: <p>In one training loop, I am reading about 100000 feature files and each file’s size is 400KB. This is slow when the files are stored on a HDD, so I added <span class="mention">@lru_cache</span> annotation for the function reading feature files. However, it seems that in each iteration of DataLoader object, new worker processes will be created (if num_workers &gt; 0), so the cached data in old worker processes is useless in the next iteration.</p>
<p>Is there a way to reuse the created worker processes, or is there an option for caching in DataLoader object? What is the best practice to read a huge amount of small sized files with PyTorch?</p>

ptrblck: <p>Have a look at <a href="https://github.com/ptrblck/pytorch_misc/blob/master/shared_array.py">this small example</a> on how to use shared arrays for multiple workers and let me know, if that works for you.</p>

wentian_zhao: <p>The thinking of using multiprocessing module to share objects between worker processes indeed works, thank you.</p>
<p>As the feature files I used has a huge total size, and cannot be identified simply with index, I used modified <code>pylru.lrucache</code> object, registered it in <code>multiprocessing.managers.BaseManager</code> and shared the cached content between processes. I’ll post the example later.</p>

andrewnaguib: <p>I’m not sure how to use your snippet. It has two questionable lines.</p>
<p>Shouldn’t the <code>use_cache</code> boolean value be reversed (i.e., initialized with True instead)</p>
<p>And <code>loader.dataset.set_use_cache(True)</code>, how are you executing it after the data looping?</p>

ptrblck: <p>I don’t think so. If you set it to <code>False</code>, the cache will be populated with your tensors.<br>
Passing it as <code>True</code> will instead get the data from the internal “cache”.<br>
Probably I should change the if-statement for better readability. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
<p>After The first epoch, I call <code>loader.dataset.set_use_cache(True)</code> so that the following epochs will just get the data from the already cached tensors.</p>

andrewnaguib: <p>Yeah, thank you, that’s much better. By the way, how can I also store the <code>x</code> length in the same <code>shared_array[index]</code> or to generalize the question, how can I store multiple values?</p>

ptrblck: <p>A more general version might be the usage of a <a href="https://github.com/ptrblck/pytorch_misc/blob/master/shared_dict.py" rel="nofollow noopener">shared dict</a>. Otherwise you could need to somehow pack the additional information into your array.<br>
Would that work for you?</p>

kevalmorabia97: <p>Why do we first create a mp.Array and then convert it to numpy array? Can’t we directly create a numpy array to create a torch.Tensor from it?</p>

ptrblck: <p>This use case is more or less an edge case and we are creating the <code>multiprocessing.Array</code> so that we can share it between multiple processes.</p>
<p>If you don’t need this functionality, you are fine using standard numpy arrays and convert them to tensors via <code>torch.from_numpy</code>.</p>

kevalmorabia97: <p>Thanks for the quick reply!<br>
Is there any advantantage of using multiprocessing.Array? over a standard numpy array and allowing multiple workers to access the data at the index they are trying to access?</p>

ptrblck: <p>The <a href="https://discuss.pytorch.org/t/reuse-of-dataloader-worker-process-and-caching-in-dataloader/30620">initial post</a> describes this special use case.</p>
<p>You would basically load the data once and reuse it for all workers.<br>
That’s not the standard work flow, where you are usually loading e.g. image files and apply some live data augmentation on it, so I would recommend to not use the shared array unless you really need it.</p>

woffett: <p>Hi there, are you able to share some example code? I’m in a similar situation where I want to share the lru_cache across workers. Thanks!</p>

kidd_lee: <p>Thank you for the <a href="https://github.com/ptrblck/pytorch_misc" rel="noopener nofollow ugc">pytorch_misc</a> repository.</p>
<h2>1. For the Read-write purpose:</h2>
<p>Just wonder what will happen if we replace</p>
<pre><code class="lang-auto">multiprocessing.Manager.dict
</code></pre>
<p>with common python <code>dict</code> in the <a href="https://github.com/ptrblck/pytorch_misc/blob/master/shared_dict.py" rel="noopener nofollow ugc">snippet</a>.</p>
<h2>2. For the Read-only purpose:</h2>
<p>If I can preload all data into a container in a proper way (not using dataset or dataloader), and then use the container as a cache for the dataset where the data will be <strong>read-only</strong> in rest execution. Is it still necessary to use the</p>
<pre><code class="lang-auto">multiprocessing.Manager.dict
</code></pre>
<p>or we can turn to the normal <code>dict</code>.</p>

ptrblck: <ol>
<li>
<p>I guess it won’t be working to share the same <code>dict</code>, as it would be copied. Did you try to replace it and run the code snippet?</p>
</li>
<li>
<p>If you can preload the data, you could use it directly in a <code>Dataset</code> and the <code>DataLoader</code> would then create copies of it for each worker.</p>
</li>
</ol>

kidd_lee: <ol>
<li>I tried to replace the <code>manager.dict</code> with a python standard dict. Nothing wired happens for the snippet.</li>
<li>I printed the <strong>object id</strong> in <code>__getitem__</code> and only observe a unique id for both two cases. Guessing dict is not duplicated in my cases.</li>
</ol>
<p>The fear is that it works ok in a small snippet but breaks in complicated circumstances.</p>
<p>I have long been wondering the mechanism of our pytorch dataloader and dataset but still get confused after searching for <a href="https://github.com/pytorch/pytorch/issues/2052#issuecomment-314692434" rel="noopener nofollow ugc">informations</a>.</p>
<p>According to the above github links, the <strong>fork/spawn</strong> methods might be the keys to dive into the details but this is diffcult for us users.</p>
<p>It is hoped that a in-depth tuitorial could be proposed on the dataset/dataloader mechenism. That would be helpful for users with a slightly advanced purpose.</p>