Caching and sharing long chunked files

username: phtephanx
cooked: <h4>Problem Statement</h4>
<p>Given is a <em>corpus of large files with multiple non-overlapping chunks</em> (i.e. samples) per file.<br>
The corpus is <em>too large to be pre-loaded completely</em>.</p>
<p>Since <code>DataLoader</code> relies on lazy loading, each time a sample is requested, the entire file is loaded, the corresponding chunk extracted, then the file discarded. To remedy this inefficiency, cache file’s signal.</p>
<p>But, how to fit caching (sharing state) seamlessly with <code>DataLoader</code> or multiprocessing in general?</p>
<h4>Attempts</h4>
<ol>
<li>Chunking offline: not desired</li>
<li>Pre-Loading corpus partially, split epoch into sub-epochs</li>
</ol>
<ul>
<li>not nice</li>
<li>reduces randomness</li>
</ul>
<ol start="3">
<li>Sharing memory</li>
</ol>
<ul>
<li>need to write to shared memory for loading to cache (slow)</li>
</ul>
<ol start="4">
<li>Splitting corpus, using <code>DistributedSampler</code> and <code>DistributedDataParallel</code> with 0 workers</li>
</ol>
<ul>
<li>avoiding shared memory</li>
<li>sampling to cure loss of randomness</li>
</ul>
<p>Is there another more convenient way of handling this?</p>

username: thiagocrepaldi
cooked: <p>The new ChunkDataset API might help you!</p>
<p>The way it works is through a hierarchical sampling, meaning it splits the dataset in chunks (set of examples) which are shuffled. Each chunk has its samples shuffled too (second layer of shuffling). C++ or Python Dataloader will retrieve batches from an internal buffer that holds just a few chunks, not the whole corpus.</p>
<p>In order to use it, all you need to do is to implement your own C++ ChunkDataReader, which parses a single chunk of data. This new chunk datareader is then passed to ChunkDataset, which will handle all the shuffling and chunk by chunk loading for you.</p>
<p>Look at an test example at: DataLoaderTest::ChunkDataSetGetBatch</p>
<p>Currently there is only C++ support, but python bindings are on the way (<a href="https://github.com/pytorch/pytorch/pull/21232" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/21232</a>) and any feedback is welcome</p>

username: phtephanx
cooked: <p>Very Interesting!</p>
<p>Let’s say, we have a data set of N chunks each of which has M samples. N * M samples in total.<br>
When buffering L chunks, I assume, the <code>ChunkDataset</code> has access to all L * M samples of the current buffer? After drawing batches from the buffer, the memory of the corresponding samples is freed, no matter which chunk the originate from? And as soon as there’s enough free memory, the next chunk is loaded to the buffer?</p>

username: thiagocrepaldi
cooked: <p>ChunkDataset will cache a few random chunks (n &lt; N) in memory and create minibatches out of them. The internal cache is continuously replenished in background, in a way there is always available data for the user, but without loading the whole thing in memory</p>
<p>This is useful in a distributed computing scenario, where each worker will load only part of the dataset</p>

username: thiagocrepaldi
cooked: <p>sorry for the delay, I didnt get notified of you reply</p>