Memory management DataLoader, optimization and realted stuff

Jere_Poveda_Martinez: <p>Hi,</p>
<p>I’ve been using PyTorch (Lightning) almost for a year. However, I have little knowledge about CS things (processes, threads, etc.).</p>
<p>Since now, my way of optimizing training time is None and my reasoning is as simple as: more data = more time, more parameters = more time. I wolud like to know how pytorch works with a bit more detail so I can use it optimally, any recomendation for this is welcome. Following this idea,  I wolud like to know more details about some things.</p>
<ul>
<li>In GPU training, the dataloader fetches the data from disk, RAM or from where?</li>
<li>If the data is transformed, for example with <a href="https://monai.io/" rel="noopener nofollow ugc">MONAI</a>, <a href="https://torchio.readthedocs.io/transforms/preprocessing.html#tocanonical" rel="noopener nofollow ugc">torchIO</a> ot torchvision, this transformations happen every epoch?</li>
<li>If I transfer data between devices in preprocessing steps with things like <code>monai.transforms.ToDevice</code>, which is basically an enhaced wrapper for <code>to()</code> this will be time-consuming?</li>
<li>In multi-GPU training, do I need to adjust my batch size so batches can fit in each GPU or the batch is splitted across GPUS and I can make it bigger?</li>
<li>Is there any way to visualize or keep track of the processes created by dataloader? The motivation for this question is that if is set <code>num workers=8</code> I wolud like to see the eight processes in something like Linux <code>htop</code>.</li>
<li>Don’t kill me for asking that, what is <code>rank_zero</code>  and why should I know more about it?</li>
<li>What is automatic mixed precision and why should I use it/know more about it?</li>
</ul>
<p>PD: any reference, resource, eplanation, exercise or advice for optimizing training is welcome, thanks.</p>

ptrblck: <p>Generally, I would recommend to take a look at the <a href="https://pytorch.org/tutorials/recipes/recipes/tuning_guide.html">performance guide</a> if you are trying to optimize your code.</p>
<p>I’ll also try to answer the questions in a general PyTorch setup, as I’m not deeply familiar with Lightning and guess that it might try to optimize the code for you in some of their utils.</p>
<aside class="quote no-group" data-username="Jere_Poveda_Martinez" data-post="1" data-topic="153767">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jere_poveda_martinez/48/50422_2.png" class="avatar"> Jere_Poveda_Martinez:</div>
<blockquote>
<p>In GPU training, the dataloader fetches the data from disk, RAM or from where?</p>
</blockquote>
</aside>
<p>It depends on the <code>Dataset</code> implementation and how the data is loaded in the <code>__init__</code> and <code>__getitem__</code>. E.g. <code>ImageFolder</code> will lazily load each sample from the specified image folders in the <code>__getitem__</code> to avoid preloading all samples (which might not even fit into your RAM).<br>
Small datasets, such as <code>MNIST</code>, are loaded into the RAM in the <code>__init__</code> method and each sample is then just indexed and transformed in the <code>__getitem__</code>.</p>
<aside class="quote no-group" data-username="Jere_Poveda_Martinez" data-post="1" data-topic="153767">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jere_poveda_martinez/48/50422_2.png" class="avatar"> Jere_Poveda_Martinez:</div>
<blockquote>
<p>If the data is transformed, for example with <a href="https://monai.io/">MONAI</a>, <a href="https://torchio.readthedocs.io/transforms/preprocessing.html#tocanonical">torchIO</a> ot torchvision, this transformations happen every epoch?</p>
</blockquote>
</aside>
<p>In the standard setup the transformation is applied in the <code>__getitem__</code> of your <code>Dataset</code> for each sample. This means that each sample in the batch will be loaded and transformed on the fly during the entire epoch.</p>
<aside class="quote no-group" data-username="Jere_Poveda_Martinez" data-post="1" data-topic="153767">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jere_poveda_martinez/48/50422_2.png" class="avatar"> Jere_Poveda_Martinez:</div>
<blockquote>
<p>which is basically an enhaced wrapper for <code>to()</code> this will be time-consuming?</p>
</blockquote>
</aside>
<p>It depeneds what you mean by “time consuming”. Pushing the data to the GPU will of course take time. However, compared to the actual model training it might be tiny and if you are using pinned memory and an async copy, the CPU would be non-blocking.</p>
<aside class="quote no-group" data-username="Jere_Poveda_Martinez" data-post="1" data-topic="153767">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jere_poveda_martinez/48/50422_2.png" class="avatar"> Jere_Poveda_Martinez:</div>
<blockquote>
<p>In multi-GPU training, do I need to adjust my batch size so batches can fit in each GPU or the batch is splitted across GPUS and I can make it bigger?</p>
</blockquote>
</aside>
<p>In the recommended approach (single process per device using <code>DistributedDataParallel</code>) each process is responsible for the data loading and will thus use its own <code>batch_size</code>. To avoid loading the same samples, a <code>DistributedSampler</code> is usually used.</p>
<aside class="quote no-group" data-username="Jere_Poveda_Martinez" data-post="1" data-topic="153767">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jere_poveda_martinez/48/50422_2.png" class="avatar"> Jere_Poveda_Martinez:</div>
<blockquote>
<p>Is there any way to visualize or keep track of the processes created by dataloader? The motivation for this question is that if is set <code>num workers=8</code> I wolud like to see the eight processes in something like Linux <code>htop</code>.</p>
</blockquote>
</aside>
<p>Yes, the processes should be visible in <code>htop</code> and <code>ps auxf</code>.</p>
<aside class="quote no-group" data-username="Jere_Poveda_Martinez" data-post="1" data-topic="153767">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jere_poveda_martinez/48/50422_2.png" class="avatar"> Jere_Poveda_Martinez:</div>
<blockquote>
<p>Don’t kill me for asking that, what is <code>rank_zero</code> and why should I know more about it?</p>
</blockquote>
</aside>
<p>I don’t know where <code>rank_zero</code> is coming from but usually <code>rank</code>s are used in a distributed trainig setup.</p>
<aside class="quote no-group" data-username="Jere_Poveda_Martinez" data-post="1" data-topic="153767">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jere_poveda_martinez/48/50422_2.png" class="avatar"> Jere_Poveda_Martinez:</div>
<blockquote>
<p>What is automatic mixed precision and why should I use it/know more about it?</p>
</blockquote>
</aside>
<p>Automatic mixed precision can speedup your training and save memory especially if you are using “newer” GPUs with TensorCores. Check the <a href="https://pytorch.org/tutorials/recipes/recipes/amp_recipe.html">amp tutorial</a> and the performance guide for more information.</p>

Jere_Poveda_Martinez: <p>Hey <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>Sorry for the delay, just one detail <code>rank_zero</code> is mentioned several times in Lightning documentation, maybe I could ask someone in their Slack to tell me a little more in depth about it (and read the docs again with a little bit of calm). I dont come from a CS background and I feel lost sometimes with so many new concepts… So I really appreciate your answer. Just one more general question, do you think that geting familiar with C++/CUDA and Operating Systems can be beneficial to me as researcher/developer in general?</p>
<p>Thaks, Yere.</p>

ptrblck: <p>Yes, I think any computer science knowledge is beneficial for researchers as well and especially understanding how the hardware works could help in isolating bottlenecks etc.</p>