How are the images shuffled within the batches?

username: PabloRR100
cooked: <p>Hi all,</p>
<p>I was wondering how the DataLoader handle the shuffling of images.</p>
<p><strong>Scenario</strong><br>
Let’s say my batch size is 64. I have a dataset of 64.000 training images, therefore, my <code>training_dataloader</code> will create 1.000 batches, each of which will contain 64 images.</p>
<p>I understand that, the moment of selecting which batch comes next in the training loop picks a random index among all the batches. <strong>Is this right?</strong></p>
<p><strong>Question</strong><br>
Does each of those 1.000 batches always contains the same 64 images or every time each single batch samples 64 images randomly?</p>
<p>Thank you very much in advance</p>

username: rasbt
cooked: <p>I think the DataLoader just shuffles the complete dataset (i.e., creates a index array of range [0, num_examples] and shuffles this) at each epoch. Then it sweeps over these indices in sequential order. E.g., if you have a dataset like [0, 1, 2, 3, 4, 5, 6] with batch size 2, a random order could be [4, 2, 1, 6, 0, 3, 5], and the minibatches would then be<br>
[4, 2], [1, 6], [0, 3], [5].</p>
<p>This way, you would have random sampling WITHOUT replacement.</p>
<blockquote>
<p>I understand that, the moment of selecting which batch comes next in the training loop picks a random index among all the batches.  <strong>Is this right?</strong></p>
</blockquote>
<p>This would be random sampling WITH replacement. I am 99.9% sure that this is not what the DataLoader does (although it is also a valid approach; actually, it is even more “correct” if you think of stochastic gradient descent; as far as I know, it is less common though and maybe not working so well empirically; with large datasets sizes, e.g., &gt; 500k, i doubt you will notice any difference in the resulting model)</p>

username: ptrblck
cooked: <p>You are absolutely correct. If no <code>sampler</code> was passed to the <code>DataLoader</code> and <code>shuffle=True</code> was set, a <a href="https://github.com/pytorch/pytorch/blob/e2d3a3fd6a248a788e3d548bb1caff9019c585ef/torch/utils/data/dataloader.py#L105" rel="nofollow noopener">RandomSampler</a> will be used.<br>
The <code>replacement</code> argument is set to <code>False</code> by default, so basically <a href="https://github.com/pytorch/pytorch/blob/e2d3a3fd6a248a788e3d548bb1caff9019c585ef/torch/utils/data/sampler.py#L75" rel="nofollow noopener">this line of code</a> will be executed as you’ve described.</p>

username: rasbt
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="3" data-topic="36466">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>The  <code>replacement</code>  argument</p>
</blockquote>
</aside>
<p>Nice, I didn’t now that this existed!</p>

username: PabloRR100
cooked: <p><strong>Scenario:</strong><br>
The question comes from the fact that I am training a ResNet on CIFAR10 and the training loss reaches 0% at around epoch 100 and I still don’t match enough accuracy on the test set.</p>
<p>I don’t think I am overfitting since the validation loss keeps reducing as well. So I didn’t go for increasing the weight decay.</p>
<p><strong>Question:</strong><br>
Any idea on how could I manage to kind of keep room for improvement and not reach the 100% accuracy while training so fast? <a class="mention" href="/u/rasbt">@rasbt</a> <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>I thought in increasing the shuffling but given your answer this is not an option any more <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=6" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"></p>
<p>Thank you!</p>

username: rasbt
cooked: <p>one of the many things to try is data augmentation (some random rotation &amp; translation, for example)</p>

username: PabloRR100
cooked: <p>I am following the ResNet paper settings for CIFAR10.<br>
I think I give way more insights of what could be happening in <a href="https://discuss.pytorch.org/t/shape-of-the-validation-loss/36597">this new question</a> after I looked at my validation loss.</p>
<p>What is the meaning of the shape?<br>
Could that be the problem why I can’t reach more than 92% of testing accuracy?</p>

username: amirsina_torfi
cooked: <p>Thank you for your explanation. Is there any working example regarding the usage of Dataloader WITH replacement?</p>

username: ptrblck
cooked: <p>I think the easiest way would be to create a <code>Random(Weighted)Sampler</code> and just pass it to the <code>DataLoader</code>. I would recommend to implement the replacement logic in the <code>sampler</code>, not the <code>Dataset</code> directly.</p>

username: amirsina_torfi
cooked: <p>Thank you for your response. I understand the logic. However, a working example can make things very easy. Do you have or saw any such working example?</p>

username: ptrblck
cooked: <p>Here is a small dummy example:</p>
<pre><code class="lang-python">data = torch.randn(10, 3, 224, 224)
target = torch.arange(10)

dataset = TensorDataset(data, target)
sampler = torch.utils.data.sampler.RandomSampler(
    dataset,
    replacement=True
)

loader = DataLoader(
    dataset, 
    batch_size=1,
    sampler=sampler
)

for data, target in loader:
    print(target)
</code></pre>
<p>Let me know, if you need more information! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: amirsina_torfi
cooked: <p>Thank you so much. Yes, I believe that can solve my problem. I recently developed a similar working example and based on your answer I believe I am on the right track!</p>