What are the (dis) advantages of persistent_workers

Rakshit_Kothari: <p>Hello,</p>
<p>What does <code>persistent_workers</code> do? What are the implications on I/O speed and RAM consumption?</p>

albanD: <p>Hi,</p>
<p>With this option to false, every time your code hits a line line <code>for sample in dataloader:</code>, it will create a brand new set of workers to do this loading and will kill them on exit.<br>
Meaning that if you have multiple dataloaders, the workers will be killed when you are done with one instantly.</p>
<p>If you make them persist, these workers will stay around (with their state) waiting for another call into that dataloader.</p>
<p>Setting this to True will improve performances when you call into the dataloader multiple times in a row (as creating the workers is expensive). But it also means that the dataloader will have some persistent state even when it is not used (which can use some RAM depending on your dataset).</p>

enterthevoidf22: <p>some follow-up questions<br>
does setting persistent workers to True cancel the re-shuffling of the dataloader each epoch?</p>
<p>specific to me:<br>
i’m running a heavy training protocol (big 3D input samples) but my dataloading is quite straight-forward, one dataloader for training and one for validation. they are re-initialized (with <code>for sample in dataloader:</code>) every epoch and i notice they take some time, sometimes up to 10 seconds (when ran in debugger) - do you think in this case it is correct to set persistence to true?</p>

albanD: <aside class="quote no-group" data-username="enterthevoidf22" data-post="3" data-topic="102110">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/enterthevoidf22/48/29338_2.png" class="avatar"> enterthevoidf22:</div>
<blockquote>
<p>does setting persistent workers to True cancel the re-shuffling of the dataloader each epoch?</p>
</blockquote>
</aside>
<p>No it should not change how randomness is done.</p>
<p>If the persistent state is not an issue for you, you should definitly enable it yes.</p>

IamHuijben: <aside class="quote no-group" data-username="albanD" data-post="4" data-topic="102110">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>If the persistent state is not an issue for you, you should definitly enable it yes.</p>
</blockquote>
</aside>
<p>Could you give an example situation in which such a persistent state of the dataloader becomes an issue? In my case I’m preloading my entire dataset in RAM and use enumerate(dataloader) to extract mini-batches.<br>
Would it then be wise to use the persistent state option?</p>

albanD: <p>It will be an issue if you use the dataloader only once or have a large number of them that you rotate and so you don’t want to have all of them with their state in memory at the same time.<br>
But if you have a single one in your case, you should definitely use it yes.</p>

BramVanroy: <p>I can imagine that if you just have a preprocessed dataset where you just need to access its <code>__getitem__</code> from within memory, that one would rather set num_workers=0 and persistent workers=False. The overhead of multiple workers will not contribute anything in that case because no processing needs to be done except for collating which I am not sure is very expensive.</p>
<p>Does that make sense, or am I wrong?</p>

albanD: <p>If you’re using <code>num_workers=0</code>, there are no worker processes, so the persistent worker flag will have no effect at all <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
But indeed, if your dataset is completely in memory and you don’t have heavy preprocessing, then you can use 0 workers and it will run just fine.</p>

Felipe_Mello: <p><a class="mention" href="/u/alband">@albanD</a> , is it possible to shut down the workers later on? I have a situation where I want to have persistent workers, but after I am done with dataloader X, I want all of its footprint to go away. I am not sure if del dataloader fixes it. Any ideas? Thanks!</p>

ejguan: <p>Yes, it is possible. When you enable <code>persistent workers</code>, you need to <code>del dl. _iterator</code> or <code>dl._iterator. _shutdown_workers()</code> to remove it from your processes.</p>

wmpauli: <p><a class="mention" href="/u/alband">@albanD</a> , thank you for answering the thread. I’m especially glad to hear that <code>persistent_workers</code> allows workers to stick around while iterating over the dataset (e.g. <code>for image, target in dataloader</code>). From the documentation, I was under the impression that workers stick around until the <em>entire</em> dataset is consumed once:</p>
<pre><code class="lang-auto">        persistent_workers (bool, optional): If ``True``, the data loader will not shutdown
            the worker processes after a dataset has been consumed once. This allows to
            maintain the workers `Dataset` instances alive. (default: ``False``)
</code></pre>

ytwanghaoyu: <p>How can I enable persistent_workers but also able to change worker’s attribute?<br>
For example, in my dataset, I have an attribute called epoch, which means what epoch it is right now, I update this attribute while training.<br>
But since I set persistent_workers to True, the worker will be the same all the time with epoch equal to the initialization value. How can I change this behavior without turning persistent_workers to False?</p>

ytwanghaoyu: <p>In case anybody has the same situation as I am, here is the solution:<a href="https://github.com/pytorch/pytorch/issues/97326" class="inline-onebox" rel="noopener nofollow ugc">Change worker process's attribute while set persistent_worker = True · Issue #97326 · pytorch/pytorch · GitHub</a></p>

daniel347x: <p>Just wanted to throw out my own use-case and why it is valuable (and a time-saver) for me to actually have persistent_workers set to False.</p>
<p>The reason is that I have an “active learning” process where the dataset labels can change every epoch.</p>
<p>All of the samples themselves are cached locally in shared memory, but the LABELS are cleaned up as active learning proceeds so these labels need to be pulled down from an external database. This pulling of the new labels occurs once per epoch, and is a relatively fast operation, a call to the database. This call is triggered by calling a function on the Dataset owned by the DataLoader.</p>
<p>By setting persistent_workers to False, it guarantees that a NEW instance of the Dataset is pickled out to every worker at the start of the epoch, based on the current instance of the Dataset in the DataLoader, with the updated labels from the database.</p>
<p>This would not be possible with persistent_workers set to True (at least without digging into the internals, which I’d rather not do).</p>