Implementing Dynamic Data Sampling for BERT Language Model Training with PyTorch DataLoader

username: Ali_H_Ahmad
cooked: <p>I’m currently in the process of building a BERT language model from scratch for educational purposes. While constructing the model itself was a smooth journey, I encountered challenges in creating the data processing pipeline, particularly with an issue that has me stuck.</p>
<p><strong>Overview:</strong></p>
<p>I am working with the IMDB dataset, treating each review as a document. Each document can be segmented into several sentences using punctuation marks (<code>.</code> <code>!</code> <code>?</code>). Each data sample consists of a <code>sentence A</code>, a <code>sentence B</code>, and an <code>is_next</code> label indicating whether the two sentences are consecutive. This implies that from each document (review), I can generate multiple training samples.</p>
<p>I am utilizing PyTorch and attempting to leverage the <code>DataLoader</code> for handling multiprocessing and parallelism.</p>
<p><strong>The Problem:</strong></p>
<p>The <code>__getitem__</code> method in the Dataset class is designed to return a single training sample for each index. However, in my scenario, each index references a document (review), and an undefined number of training samples may be generated for each index.</p>
<p><strong>The Question:</strong></p>
<p>Is there a recommended way to handle such a situation? Alternatively, I am considering the following approach:</p>
<p>For each index, an undefined number of samples are returned to the <code>DataLoader</code>. The <code>DataLoader</code> would then assess whether the number of samples is sufficient to form a batch. Here are the three possible cases:</p>
<ol>
<li>The number of samples returned for an index is less than the batch size. In this case, the <code>DataLoader</code> fetches additional samples from the next index (next document), and any excess is retained to form the next batch.</li>
<li>The number of samples returned for an index equals the batch size, and it passes it to the model.</li>
</ol>
<p>I appreciate any guidance or insights into implementing this dynamic data sampling approach with PyTorch <code>DataLoader</code>.</p>

username: JuanFMontesinos
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/2/922d9cbdf418ff2ad1dab183e6d60eca2f7ed86b.png" data-download-href="https://discuss.pytorch.org/uploads/default/922d9cbdf418ff2ad1dab183e6d60eca2f7ed86b" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/2/922d9cbdf418ff2ad1dab183e6d60eca2f7ed86b.png" alt="image" data-base62-sha1="kR9qWkQpAkT3SyL5fAjUg8HWS2T" width="690" height="493" data-dominant-color="F5F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">874×625 49.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
You can load a batch directly in <strong>getitem</strong> with the proper config of the dataloader</p>

username: Ali_H_Ahmad
cooked: <p>Thank you very much for your interest,</p>
<p>Are you suggesting to disable automatic batching and take control of the batch formation in my training loop, something like the following?</p>
<pre><code class="lang-auto">my_dataset = MyDataset(data)
data_loader = DataLoader(my_dataset, batch_size=None, batch_sampler=None)

batch_size = 32
current_batch = []

# The data_loader may return one or more samples ..
for samples in data_loader:
    samples = process_samples(samples)

    # Accumulate samples until reaching the desired batch size
    # ...
    current_batch.append(samples)
    # ...

    if len(current_batch) == batch_size:
        processed_batch = process_batch(current_batch)

        # Forward pass, backward pass, and optimization steps ...

        current_batch = []

</code></pre>
<p>Thanks again <img src="https://discuss.pytorch.org/images/emoji/apple/innocent.png?v=12" title=":innocent:" class="emoji" alt=":innocent:" loading="lazy" width="20" height="20"></p>

username: JuanFMontesinos
cooked: <p>Hi,<br>
when you disable the automatic batching, you can return a batch directly from the <strong>getitem</strong> or, as you show in the script, decide when your batch size is large enough.<br>
The idea is you manually conform your batch:</p>
<p>and an undefined number of training samples may be generated for each index.</p>
<p>I think it would be more optimized if you can do everything in <strong>getitem</strong> but I imagine it’s harder that way.</p>

username: Ali_H_Ahmad
cooked: <p>I’ve thought about it, but it’s really difficult, so I guess I’ll just stick to the method I showed you. I think it would not be bad if the Batch size was small (such as 32).</p>