How to create hierarchical datasets/dataloaders?

Noam_Salomonski: <p>Hi, I am asking here because it seemed like the right place, if it isn’t please tell me where to ask.</p>
<p>Consider a stream of tabular data.</p>
<pre><code class="lang-auto">import pandas as pd
import numpy as np


def data_stream():
    for _ in range(1000):
        df = pd.DataFrame({
            'a': np.arange(10000),
            'b': (np.arange(10000) + 10000)
        })
        yield df
</code></pre>
<p>Please assume the dataframes will be large (and different).</p>
<p>I want to create a dataloader for data that is arranged as I stated above.<br>
batches should be of X rows of the current dataframe, until it is done (including shuffling flexibility ect.). Can throw away the last batch if it is not full.<br>
Then, go on to the next dataframe, until StopIteration.</p>
<p>If it were a single dataframe, I would simply use the good old torch.utils.data.Dataset with a standard dataloader, with small configuration of the number of df rows per sample and be done.</p>
<p>If it were a stream of single sample per stream item, I would use torch.utils.data.IterableDataset exactly like the doc states.</p>
<p>However, I have both.</p>
<p>If I use a torch.utils.data.IterableDataset, I have to define a DataLoader for it, and I then lose the power of the DataLoader that would operate on the df itself. The same problem would arise in the other direction.</p>
<p><a href="https://github.com/pytorch/pytorch/issues/50139" rel="noopener nofollow ugc">was move from here</a></p>

Noam_Salomonski: <p>What’s the acceptable behavior here about bumping posts?</p>

nairbv: <p>You might find the recent discussions on composable datasets useful or interesting.</p>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/49440" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/49440" target="_blank" rel="noopener nofollow ugc">[RFC] DataLoader architecture updates and TarDataset implementation</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2020-12-16" data-time="00:34:47" data-timezone="UTC">12:34AM - 16 Dec 20 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/VitalyFedyunin" target="_blank" rel="noopener nofollow ugc">
          <img alt="VitalyFedyunin" src="https://avatars2.githubusercontent.com/u/1163042?v=4" class="onebox-avatar-inline" width="20" height="20">
          VitalyFedyunin
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">Problem statement
This proposal aims to construct a modular, user-friendly, and performant toolset to address the ambiguous activity referred to as “dataloading”...</p>
</div>

<div class="labels">
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">feature</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">module: dataloader</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">triaged</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>