Dataloader for partitioned dataset

username: iknunyants
cooked: <p>I have a large dataframe dataset. There are float columns but also columns with lists of int. I’ve partitioned it into smaller files.  Tried custom dataset which gets [filename, index] in getitem and loads this file if not already loaded. But when I set num_workers &gt; 1, ram used by processes keeps increasing sufficiently. What is the efficient way to construct dataloader which can read files one by one?</p>

username: ejguan
cooked: <p>Could you share code snippet?</p>

username: iknunyants
cooked: <p>Here is my code in custom sampler. In self.indices[i][j], i - parquet file, j - set of indicies</p>
<pre><code class="lang-auto">def __iter__(self):
    all_batches = []
    for j, file in enumerate(self.datafiles):
        batches = []
        for inds in self.indices[j]:
            sampler = BatchSampler(SubsetRandomSampler(inds), batch_size=self.batch_size, drop_last=self.drop_last)
            for sample in sampler:
                batches.append(sample)
            shuffle(batches)
            all_batches.extend(batches)
    for batch in all_batches:
        yield batch
</code></pre>
<p>Here is what I do in my custom dataset getitem</p>
<pre><code class="lang-auto">def __getitem__(self, idx: Tuple[str, int]) -&gt; Tuple:
    file, idx = idx
    if self.cur_file != file:
        self.cur_file = file
        del self.data
        self.data = pd.read_parquet(self.cur_file)
        gc.collect()
    credit = self.data.iloc[idx].copy()
</code></pre>

username: ejguan
cooked: <p>I am curious about the size of each parquet file.<br>
When you run multiple workers, the RAM usage would be <code>num_of_workers</code> * <code>parquet_size</code> in total. In order to reduce the amount of memory, either reduce <code>num_of_worker</code> or reduce <code>parquet_size</code> would be the best way.</p>

username: iknunyants
cooked: <p>I agree, this is one way. But are there any solutions. Alternatives to pandas dataframe? Other structure of  my custom dataset? I’m open to your ideas <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=12" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:" loading="lazy" width="20" height="20"></p>

username: ejguan
cooked: <p>IIUC, We currently do have a solution to stream parquet file using <code>TorchData</code>. But, it requires you to run pipeline in Iterative-style using <code>DataPipe</code>.<br>
See: <a href="https://github.com/pytorch/data/blob/b6ade8f097bc9ac08460cd403034a35daff09cfa/torchdata/datapipes/iter/util/dataframemaker.py#L82-L83" class="inline-onebox" rel="noopener nofollow ugc">data/dataframemaker.py at b6ade8f097bc9ac08460cd403034a35daff09cfa · pytorch/data · GitHub</a></p>
<p>cc: <a class="mention" href="/u/nivek">@nivek</a></p>

username: ejguan
cooked: <aside class="quote no-group" data-username="iknunyants" data-post="3" data-topic="149777">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/iknunyants/48/48842_2.png" class="avatar"> iknunyants:</div>
<blockquote>
<pre><code class="lang-auto">    credit = self.data.iloc[idx].copy()
</code></pre>
</blockquote>
</aside>
<p>Besides, I am curious about <code>credit</code> object. Do you de-reference it later in your training loop?</p>

username: iknunyants
cooked: <p>I do, yes. I’m taking column values from it.<br>
Thanks for mentioning TorchData, I’ll try it later</p>

username: Dushan
cooked: <p>This seems to be the best way to handle large partitioned datasets that I can find after a lot of searching.</p>
<p>Im surprised there isn’t a native solution in pytorch, or more examples out there, since I thought NLP datasets would be too big for memory by default, and text doesn’t fit into the vision framework of seperate files for each training sample.</p>
<p>Is there something I’m missing? Do I really have to write code my self to track (file, index). Surely there’s a library that abstracts this detail away for me.</p>