Slower iterations with larger datasets

Ricu: <p>Hello everyone,</p>
<p>I am working with the glue dataset and am using the <code>run_glue_no_trainer.py</code> script from the transformer github page (<a href="https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification" rel="noopener nofollow ugc">LINK</a>).</p>
<p>I am using BERT-base with a batch size of 32 on a single GPU (using standard finetuning setting from BERT paper). When working with the CoLA dataset (~8.5k train samples) I get around 10/iterations per second. When switching to the QNLI dataset (~100k train samples) I only get around 5/iterations per second.</p>
<p>This slowdown is significant and even more noticeable with larger models. I would assume the iteration/s to stay approximately stay the same as long as only the dataset is swapped out.</p>
<p>My hardware setup is as follows:<br>
CPU Intel(R) Xeon(R) Silver 4310 CPU @ 2.10GHz<br>
GPU NVIDIA A40<br>
RAM: 64GB</p>

ptrblck: <aside class="quote no-group" data-username="Ricu" data-post="1" data-topic="192025">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ricu/48/58380_2.png" class="avatar"> Ricu:</div>
<blockquote>
<p>This slowdown is significant and even more noticeable with larger models. I would assume the iteration/s to stay approximately stay the same as long as only the dataset is swapped out.</p>
</blockquote>
</aside>
<p>Not necessarily as it would depend on the data processing, and if it could create a bottleneck, as well as the sample loading, which could also create a blottleneck e.g. if you have more random accesses from a spinning HDD.<br>
You could profile the data loading and check if it is indeed creating the slowdown or if it’s coming from another part of your training pipeline.<br>
<a href="https://github.com/pytorch/examples/blob/f0d6fc9909425760be85c6ec40a01bc9544a1e3b/imagenet/main.py#L318">Here</a> is a simple example showing how the <code>DataLoader</code> is times in the ImageNet example.</p>

Ricu: <p>I did some timing with the PytorchProfiler (schedule with skip=10,warmup=2,active=3) to get insight into the training steps. Unfortunately I am not deep enough in the fundamentals to draw definite conclusions myself.</p>
<p>The following is one step in the training loop using CoLA dataset (~small).<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/f/5f3aba7a1ffcb4f99c4b4aed4599ac36f74e9586.png" data-download-href="https://discuss.pytorch.org/uploads/default/5f3aba7a1ffcb4f99c4b4aed4599ac36f74e9586" title="grafik"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/f/5f3aba7a1ffcb4f99c4b4aed4599ac36f74e9586_2_690x219.png" alt="grafik" data-base62-sha1="dArbseG1FeX1qsiuZN8coAVyAce" width="690" height="219" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/f/5f3aba7a1ffcb4f99c4b4aed4599ac36f74e9586_2_690x219.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/f/5f3aba7a1ffcb4f99c4b4aed4599ac36f74e9586_2_1035x328.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/f/5f3aba7a1ffcb4f99c4b4aed4599ac36f74e9586_2_1380x438.png 2x" data-dominant-color="E4E1E2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">grafik</span><span class="informations">2033×646 44.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The following is one steps in the training loop using QNLI dataset (~large).<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/7/67d9ef89d94f90854458e7400772fe2e0ff7bb3c.png" data-download-href="https://discuss.pytorch.org/uploads/default/67d9ef89d94f90854458e7400772fe2e0ff7bb3c" title="grafik"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/7/67d9ef89d94f90854458e7400772fe2e0ff7bb3c_2_690x219.png" alt="grafik" data-base62-sha1="eOI6tPn97qmeusDCgHC9wyk1i68" width="690" height="219" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/7/67d9ef89d94f90854458e7400772fe2e0ff7bb3c_2_690x219.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/7/67d9ef89d94f90854458e7400772fe2e0ff7bb3c_2_1035x328.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/7/67d9ef89d94f90854458e7400772fe2e0ff7bb3c_2_1380x438.png 2x" data-dominant-color="DCE0E4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">grafik</span><span class="informations">2041×650 41.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The step is significantly slower for QNLI than for (~240ms vs ~100ms) CoLA. Notably, the <code>aten::to</code> operation is notably slower aswell as the <code>autograd::engine::... EmbeddingBackward0</code>. The <code>enumerate(DataLoader)#_SingleProcessDataLoaderIter.__next__</code> is taking roughly the same amount of type in both cases.</p>
<p>I can provide the trace files for both runs.</p>
<p>Another thing I noted is that all steps in CoLA (~small) take roughly the same time. But for QNLI, some are shorter and some longer:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/3/83367eb0a2884711bbdd394acb35792f1a4deec5.png" data-download-href="https://discuss.pytorch.org/uploads/default/83367eb0a2884711bbdd394acb35792f1a4deec5" title="grafik"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/3/83367eb0a2884711bbdd394acb35792f1a4deec5_2_690x226.png" alt="grafik" data-base62-sha1="iILiOb8PQ4JSb5bYLuW62SqYg8l" width="690" height="226" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/3/83367eb0a2884711bbdd394acb35792f1a4deec5_2_690x226.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/3/83367eb0a2884711bbdd394acb35792f1a4deec5_2_1035x339.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/3/83367eb0a2884711bbdd394acb35792f1a4deec5_2_1380x452.png 2x" data-dominant-color="DEDEE5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">grafik</span><span class="informations">1981×650 51.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Hopefully this helps to analyze whats happening.</p>
<p>PS: Harddisk is SSD<br>
PPS: If you need the trace files, I can send them to you.</p>

Ricu: <p>Bump. Think this could be interesting to many.</p>

ptrblck: <aside class="quote no-group" data-username="Ricu" data-post="3" data-topic="192025">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ricu/48/58380_2.png" class="avatar"> Ricu:</div>
<blockquote>
<p>Notably, the <code>aten::to</code> operation is notably slower aswell as the <code>autograd::engine::... EmbeddingBackward0</code>.</p>
</blockquote>
</aside>
<p>I don’t believe that’s the case as both ops wait for a synchronization, so you would need to check what exactly is synchronizing the code.</p>

Ricu: <p>Any clue on how to find out whats synchronizing?</p>
<p>As its single-gpu I am not quite sure what kind of sources for synchronization there are.</p>

ptrblck: <p>You could profile your code with Nsight Systems and enable CPU stacktraces. Hovering over the synchronizing call would then show which operation is calling into it. Adding nvtx markers will also help in narrowing down which part of the code the sync is coming from.</p>