How does shuffle in data loader work?

username: cat-loves-donuts
cooked: <p>Well, I am just want to ask how pytorch shuffle the data set. And this question probably is a very silly question.</p>
<p>I mean I set shuffle as True in data loader. And I just wonder how this function influence the data set. For example, I put the whole MNIST data set which have 60000 data into the data loader and set shuffle as true. Does it possible that if I only use 30000 to train the model but the model cannot identify number 6 to 9 because I shuffled the data loader and all the sample of number 6 to 9 were shuffled into the last 30000 which I did not use?</p>
<p>I mean does it possible that the shuffle operation will lead to models that use partial training sets are not able to get all features of the whole data set?</p>

username: ptrblck
cooked: <p>If you set <code>shuffle=True</code>, internally the <code>RandomSampler</code> will be used, which just permutes the indices of all samples as seen <a href="https://github.com/pytorch/pytorch/blob/0b868b19063645afed59d6d49aff1e43d1665b88/torch/utils/data/sampler.py#L79" rel="nofollow noopener">here</a>.</p>
<p>Are you slicing the data before passing it to the <code>DataLoader</code>?<br>
If not, all samples should be used regardless if shuffle is active of not.</p>

username: cat-loves-donuts
cooked: <p>Thank you so much for answering this question.</p>
<p>Well, I loaded the whole data set which have 60000 data into the data loader with shuffle = True. But when I train this model, I only use like 6400 of those data. I trained the model in 100 epoch and each epoch have 64 images.</p>
<p>So I am just wondering does those 6400 data have all numbers? I means does it possible I was very lucky that only got 0 to 5 and the model could not identify the rest of numbers when it was tested?<img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=9" title=":joy:" class="emoji" alt=":joy:"><img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=9" title=":joy:" class="emoji" alt=":joy:"><img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=9" title=":joy:" class="emoji" alt=":joy:"><img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=9" title=":joy:" class="emoji" alt=":joy:">  If this happened, the accuracy should be lower…</p>
<p>But interesting thing is all my test accuracy is pretty normal. Like I trained it with 6400 data and test it on test data-set which have 10000 data and I still have 80% ~ 70% accuracy.</p>

username: ptrblck
cooked: <p>How did you use this subset of 6400 samples?<br>
Did you use a <code>Subset</code> or <code>RandomSubsetSampler</code>?<br>
Depending on the approach, you might have selected some particular classes and left out others.</p>
<p>However, if you’ve used some break statement inside the training loop for the <code>DataLoader</code> (which was supposed to use all data), it would be unlikely to just get a subset of classes.</p>

username: cat-loves-donuts
cooked: <p>Well, this is my code:<br>
def train_model(model, linear, input_CNN, learning_rate, nsamples, load_train_dataset):<br>
epoch = 0<br>
print_loss = 0<br>
criterion = nn.CrossEntropyLoss()<br>
optimizer_module = optim.SGD(model.parameters(), lr=learning_rate)<br>
optimizer_linear = optim.SGD(linear.parameters(), lr=learning_rate)<br>
optimizer_input_CNN = optim.SGD(input_CNN.parameters(), lr=learning_rate)</p>
<pre><code>for i, (data, target) in enumerate(load_train_dataset()):
    if i &gt; nsamples:
        break
    else:
        # train network
</code></pre>
<p>the parameter nsamples is 100 which means train 100 epoch. I don’t know if this code is right or not. I loaded all the data set into the data loadeer but use this nsamples to control the size of training data set.</p>

username: ptrblck
cooked: <p>This code might work and should create random batches, if <code>shuffle=True</code>.<br>
If you don’t want to use all samples, you could of course use a <code>Subset</code> and avoid using the break statement.</p>

username: cat-loves-donuts
cooked: <p>Alright, I will try to use Subset rather than this.<br>
But could you please help me one more thing? Could the way that I used make the model learn incomplete features of the training data set?</p>

username: ptrblck
cooked: <p>Could you explain a bit, what you mean by “incomplete features of the training data set”?</p>

username: cat-loves-donuts
cooked: <p>OK…<img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=9" title=":joy:" class="emoji" alt=":joy:"><img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=9" title=":joy:" class="emoji" alt=":joy:"><img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=9" title=":joy:" class="emoji" alt=":joy:"><img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=9" title=":joy:" class="emoji" alt=":joy:"><br>
My fault, I mean I used MINST data set. Does it possible that my method let the model learn 0 to 5 but missing 6 to 9? And this model can only identify 0 to 5 and show a really poor result?</p>

username: ptrblck
cooked: <p>It’s unlikely that your model has only seen samples from classes 0 to 5, if you’ve shuffled the dataset.<br>
Even though you are only using approx. 10% of the dataset, the batches should be well shuffled.</p>
<p>Here is a small example:</p>
<pre><code class="lang-python">dataset = datasets.MNIST(root='./data',
                         transform=transforms.ToTensor())
loader = DataLoader(dataset, batch_size=64, shuffle=True)

targets = []
for idx, (data, target) in enumerate(loader):
    targets.append(target)
    if idx &gt;= 99:
        break

targets = torch.cat(targets)
print(targets.shape)
&gt; torch.Size([6400])
print(targets.unique(return_counts=True))
&gt;(tensor([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]), tensor([634, 666, 657, 666, 606, 579, 629, 684, 637, 642]))
</code></pre>
<p>As you can see, each class has approx. the same number of samples, which should avoid overfitting to a subset of the classes.</p>
<p>The overfitting might be a result of another code part. Are you manipulating the data in some other way?</p>

username: cat-loves-donuts
cooked: <p>Ah ha…This is what I need. Thank you so much.<br>
Well, my results are pretty good. I mean there is no overfitting or something like that. Because there is no overfitting, I just wondering if those assumptions are valid or not…<img src="https://discuss.pytorch.org/images/emoji/apple/rofl.png?v=9" title=":rofl:" class="emoji" alt=":rofl:"><img src="https://discuss.pytorch.org/images/emoji/apple/rofl.png?v=9" title=":rofl:" class="emoji" alt=":rofl:"><img src="https://discuss.pytorch.org/images/emoji/apple/rofl.png?v=9" title=":rofl:" class="emoji" alt=":rofl:"><img src="https://discuss.pytorch.org/images/emoji/apple/rofl.png?v=9" title=":rofl:" class="emoji" alt=":rofl:"></p>
<p>But thank you so much for answering me those questions. Right now I don’t need to change my code. Thank you so much.<img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=9" title=":grin:" class="emoji" alt=":grin:"><img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=9" title=":grin:" class="emoji" alt=":grin:"><img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=9" title=":grin:" class="emoji" alt=":grin:"><img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=9" title=":grin:" class="emoji" alt=":grin:"></p>

username: saba
cooked: <p>Hi Ptrblck,</p>
<p>Just wondering to be sure that random sampling that shuffling do in the data loader is uniform? It uniformly get indices of the data .</p>

username: ptrblck
cooked: <p>If <code>shuffle=True</code>, the <code>DataLoader</code> will use a <code>RandomSampler</code> as seen <a href="https://github.com/pytorch/pytorch/blob/7494f0233aabb5109e068f3cad73e20dc6a8dd23/torch/utils/data/dataloader.py#L266">here</a>, which uses <code>torch.randperm</code> in the default setup (<code>replacement=False</code>) and randomly permutes the sample indices. There is thus no uniform sampling but a permutation of the indices.</p>