If I set torch.backends.cudnn.benchmark=True, should I also set drop_last in DataLoader?

username: frostime
cooked: <p>Hi.</p>
<p>It’s commonly said that, if one wants to set <code>torch.backends.cudnn.benchmark=True</code> so as to speed up pytorch computation, he or she should always ensure that <strong>the input size of batches stay constants</strong>.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/2/129c19083f91c2377f5ecd099004bff2d838bbae.png" data-download-href="https://discuss.pytorch.org/uploads/default/129c19083f91c2377f5ecd099004bff2d838bbae" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/2/129c19083f91c2377f5ecd099004bff2d838bbae.png" alt="image" data-base62-sha1="2ED20DvzcGdugcQZkQQfTTR6HDg" width="690" height="179" data-dominant-color="292F37"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">955×248 32.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>This arouse me a question. As we know, the size of a batch output from the <code>DataLoader</code> dose not always equal to the <code>batch_size</code> parameter we pass to it, because the dataset size is not always divisible by the <code>batch_size</code>. So we got a <code>drop_last</code> parameter in <code>DataLoader</code> to determine whether to drop the last incomplete batch.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/1/b12fa43cbc72b1a065b5937218d66d836f86d884.png" data-download-href="https://discuss.pytorch.org/uploads/default/b12fa43cbc72b1a065b5937218d66d836f86d884" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/1/b12fa43cbc72b1a065b5937218d66d836f86d884_2_690x140.png" alt="image" data-base62-sha1="phsAcdspeaIg8QFOycnseLmJrDe" width="690" height="140" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/1/b12fa43cbc72b1a065b5937218d66d836f86d884_2_690x140.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/1/b12fa43cbc72b1a065b5937218d66d836f86d884_2_1035x210.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/1/b12fa43cbc72b1a065b5937218d66d836f86d884.png 2x" data-dominant-color="E5E7EC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1095×223 26.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The default value of <code>drop_last</code> is <code>False</code>, meaning that the last batch’s size <strong>might be smaller</strong>, which violates the precondition for <code>torch.backends.cudnn.benchmark=True</code>.</p>
<p>So, should I always set <code>drop_last=True</code> when I want to set <code>torch.backends.cudnn.benchmark=True</code>?</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>Using constant input sizes is not a hard requirement, but rather a recommendation.<br>
For each new input shape cuDNN will profile available kernels internally and select the fastest one, which is then added to a cache. In the next iteration, this cache is checked for the currently used shapes and the already selected kernels will be picked.<br>
The profiling itself will add an overhead and thus your first iteration for each new shape would be slower.</p>
<p>Depending on your use case and how many different input shapes you are expecting, using <code>cudnn.benchmark=True</code> could still work.<br>
Your particular concern would rerun the profiling for the last batch if it’s smaller, so I wouldn’t bother dropping the last batch.</p>