Loading big dataset (bigger than memory) using pytorch

bkuriach: <p>I have some data which is thrice large as my system’s RAM. I need to run some Deep Learning models using pytorch. Could you please advise how can I use torch data loaders (or alternative) in this scenario?</p>
<p>Assume my data is stored as subfolders inside parent directory as below.</p>
<pre><code class="lang-auto">Transaction_Data/
              ---Customer1/
                          --- day1
                          --- day2
                          .
                          --- dayN
              ---Customer2/
                          --- day1
                          --- day2
                          .
                          --- dayN
             ---CustomerN/
                          --- day1
                          --- day2
                          .
                          --- dayN
</code></pre>
<p>Lets assume these are clean data and each customer is like an individual data frame (days represent rows).</p>
<p>I want to load these data in batches (probably 5 customers at once - can fit into memory) and train DL model using torch. What is the efficient way to load these?</p>
<p>I need to iterate over these data and run DL models.  Should I be using custom dataset? Any pointers would be really appreciated.</p>
<p>Thank you very much in advance!</p>

eqy: <p>I don’t think you need to do anything special for this scenario as usually the amount of memory required is some small multiple of the <code>batch_size</code> (due to prefetching). For example ImageNet spans hundreds of gigabytes (even more when converted to uncompressed RGB format) yet only a small fraction of this (host and device) memory is required for training.</p>

bkuriach: <p><a class="mention" href="/u/eqy">@eqy</a> Thank you for your response!<br>
I understand that we will need only small memory to fit those batches, but how will I load the data in the first place given the above folder structure?</p>

eqy: <p>It is difficult to understand what your data organization is without knowing what the meaning of <code>data1</code> and <code>data2</code> are. Are they different classes? Different datasets? If they are different classes, then a <code>DatasetFolder</code> or <code>ImageFolder</code> is exactly the abstraction for this this scenario: <a href="https://pytorch.org/vision/0.8/datasets.html#imagefolder" rel="noopener nofollow ugc">torchvision.datasets — Torchvision 0.8.1 documentation (pytorch.org)</a></p>

bkuriach: <p>No, they are not different classes. Hope below sample structure explains it better.</p>
<pre><code class="lang-auto">Transaction_Data/
              ---Customer1/
                          --- day1
                          --- day2
                          .
                          --- dayN
              ---Customer2/
                          --- day1
                          --- day2
                          .
                          --- dayN
             ---CustomerN/
                          --- day1
                          --- day2
                          .
                          --- dayN

</code></pre>
<p>Lets assume these are clean data and each customer is like an individual data frame (days represent rows).</p>
<p>I want to load these data in batches (probably 5 customers at once - can fit into memory) and train DL model using torch. What is the efficient way to load these?</p>
<p>Thank you very much in advance!</p>

eqy: <p>You can start by taking a look at the default dataset classes: <a href="https://pytorch.org/docs/stable/data.html#map-style-datasets" class="inline-onebox" rel="noopener nofollow ugc">torch.utils.data — PyTorch 1.8.1 documentation</a><br>
and seeing if your data fits the map style of iterable style abstraction. The map style is usually a straightforward abstraction for many datasets as you only need to define an <code>__getitem__</code> and a <code>__len__</code> function. Once you have a usable dataset, using a dataloader <a href="https://pytorch.org/docs/stable/_modules/torch/utils/data/dataloader.html#DataLoader" class="inline-onebox" rel="noopener nofollow ugc">torch.utils.data.dataloader — PyTorch 1.8.1 documentation</a> will handle the parallelization and loading in memory for you.</p>

bkuriach: <p><a class="mention" href="/u/eqy">@eqy</a>  Sure I will check those. Thank you very much!!</p>