Configuring Dataset to load a very large file whilst supporting multiple workers

Loua: <p>I’m having trouble creating a Dataset class that fully meets my specifications. I can imagine that others have also had this problem, but I have been unable to find a proper solution on the internet.</p>
<p>I am working with a very large (200GB+) data file which I want to dynamically load into a pytorch Dataset class. Ideally, I need the following process to happen when <strong>getitem</strong>(idx) is called:</p>
<ol>
<li>By some mechanism, the idx’th line of the datafile (in my case jsonl) is loaded into memory.</li>
<li>Some reasonably computationally expensive data augmentation is applied to the loaded data.</li>
<li>The augmented data is encoded into a torch tensor and returned.</li>
</ol>
<p>Because of (2), and due to the fact that I am using multiple GPUs to ultimately train my model, I need the Dataset to support multiple workers (in the DataLoader). I cannot express how huge the datafiles that I’m working with are. I am trying find a simple solution for (1) that ideally does not involve streaming my data in a proper DB. Here is what I have tried:</p>
<ol>
<li>
<p>Using a IterableDataset. On the surface this solution sounds like a good idea - I could directly use the readline method in the jsonlines package to create an iterator. The downside is that configuring this to work with multiple workers really sounds like a pain (each worker would be given it’s own instance of the iterator ).</p>
</li>
<li>
<p>Using mmap or something similar. One idea I had is to index the data file, recording the byte locations of the beginning of each line in my datafile. I could then use mmap.move and mmap.readline to load a particular line, by referencing it’s location using the index. This solution would probably work, but I have never worked with mmap before.</p>
</li>
</ol>
<p>Is the industry standard approach to use (2)? If so, are there any Python packages that will do the heavy lifting for me, or should I rely on mmap? I would have assumed that people working with big datasets in NLP have come across this problem.</p>
<p>Thanks in advance.</p>

nairbv: <blockquote>
<p>Is the industry standard approach to use (2)?</p>
</blockquote>
<p>I think probably the most common approach is to simply break up the file up into a larger number of smaller file partitions. Then each worker can load from different subset of the underlying files.</p>

nlgranger: <p>I’d say option 2 is more standard.<br>
With option 1 skipping to the last element of the dataset will be very slow.</p>

Loua: <p>I found the following relevant stackexchange post</p>
<aside class="onebox stackexchange" data-onebox-src="https://stackoverflow.com/questions/69784480/how-to-set-current-position-of-mmap-mmap-seekpos-to-beginning-of-any-nth-lin">
  <header class="source">

      <a href="https://stackoverflow.com/questions/69784480/how-to-set-current-position-of-mmap-mmap-seekpos-to-beginning-of-any-nth-lin" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/11814996/naveen-reddy-marthala" target="_blank" rel="noopener nofollow ugc">
    <img alt="Naveen Reddy Marthala" src="https://www.gravatar.com/avatar/43d9fc96f4ebaef1045660efdb463bc2?s=256&amp;d=identicon&amp;r=PG&amp;f=y&amp;so-version=2" class="thumbnail onebox-avatar" width="256" height="256">
  </a>

<h4>
  <a href="https://stackoverflow.com/questions/69784480/how-to-set-current-position-of-mmap-mmap-seekpos-to-beginning-of-any-nth-lin" target="_blank" rel="noopener nofollow ugc">How to set current position of `mmap.mmap.seek(pos)` to beginning of any Nth line for a text file?</a>
</h4>

<div class="tags">
  <strong>python, python-3.x, csv, mmap</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/11814996/naveen-reddy-marthala" target="_blank" rel="noopener nofollow ugc">
    Naveen Reddy Marthala
  </a>
  on <a href="https://stackoverflow.com/questions/69784480/how-to-set-current-position-of-mmap-mmap-seekpos-to-beginning-of-any-nth-lin" target="_blank" rel="noopener nofollow ugc">06:40AM - 31 Oct 21 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>I’m going to implement it using mmap and jsonl.</p>