Dataloader inside a for loop

username: SU801T
cooked: <p>Hi,</p>
<p>I was wondering if there is a significant disadvantage placing a dataloader and dict inside a for loop.</p>
<p>I am trying something where a random sample of training data is used for each epoch as I have millions of examples but not enough resources for effective training. Therefore, I want to use a bash script that randomly copies a certain amount of images from several directories and use this for training. The same is also applied for validation.</p>
<p>The pseudocode would look something like this:</p>
<pre><code class="lang-auto">
for epoch in epochs:

     run bash script to copy random number of train and validation data per epoch
     
     create dataloader dicts and dataloader
    
     do training
     do validation


    delete train and validation data 

</code></pre>
<p>Would this be ok?</p>

username: ptrblck
cooked: <p>Your workflow should generally be alright, but I would be concerned about the performance hit caused by the copies of the data. If possible, leave all data in the current structure and use a <code>Subset</code> or <code>SubsetRandomSampler</code> to load only the desired samples instead.</p>

username: SU801T
cooked: <p>Do you mean in terms of the time it takes to run the experiment or in terms of accuracy when you mention performance?</p>
<p>The issue is I am worried if it means loading an entire dataset into a dataloader or dict before the random sampling. Is this something I should be concerned about or does random sampler not even load the entire dataset and then select random samples?</p>
<p>The issue is my dataset is also incredibly imbalanced. I want to copy all of the images from one class but from the other class, I would like to select random samples.  There is a further abstraction where there are small patch images selected from entire images. I want to select a certain number of these patches from each entire image. The directory looks something like this:</p>
<pre><code class="lang-auto">
class 1 --- image 1 --- patch 1
                               --- patch 1
                               --- patch 3
                                --- patch ...

             --- image 2 --- patch 1
                               --- patch 1
                               --- patch 3
                                --- patch ...

                --- image ... --- patch 1
                               --- patch 1
                               --- patch 3
                                --- patch ...


class 2  --- image 1 --- patch 1
                               --- patch 1
                               --- patch 3
                                --- patch ...

             --- image 2 --- patch 1
                               --- patch 1
                               --- patch 3
                                --- patch ...

                --- image ... --- patch 1
                               --- patch 1
                               --- patch 3
                                --- patch ...

</code></pre>

username: ptrblck
cooked: <aside class="quote no-group" data-username="SU801T" data-post="3" data-topic="132805">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/94ad74/48.png" class="avatar"> SU801T:</div>
<blockquote>
<p>Do you mean in terms of the time it takes to run the experiment or in terms of accuracy when you mention performance?</p>
</blockquote>
</aside>
<p>I meant the speed, as moving a large dataset around might be quite expensive compared to the actual training.</p>
<aside class="quote no-group" data-username="SU801T" data-post="3" data-topic="132805">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/94ad74/48.png" class="avatar"> SU801T:</div>
<blockquote>
<p>The issue is I am worried if it means loading an entire dataset into a dataloader or dict before the random sampling. Is this something I should be concerned about or does random sampler not even load the entire dataset and then select random samples?</p>
</blockquote>
</aside>
<p>If your <code>Dataset</code> is lazily loading each sample in its <code>__getitem__</code> the recreating of <code>DataLoaders</code> using different <code>Subset</code>s or samplers should be cheap as only the <code>Dataset.__init__</code> will be called (which should not load any large amount of data if you are applying lazy loading).</p>