'shuffle' in dataloader

username: Michael-J98
cooked: <p>I noticed one strange thing that the loss value would be increased simply when I turn ‘shuffle’ off like below:<br>
<code>torch.utils.data.DataLoader(dataset_test, batch_size=batch_size, **shuffle=False**, num_workers=num_workers, drop_last=True)</code> .<br>
It’s about from 0.02 to 0.09. I didn’t change anything else but the ‘shuffle’. Anyone know what was going on? Thanks a lot!</p>

username: Nikronic
cooked: <p>Hi,</p>
<p>If I understand correctly, you only turned shuffle off and experienced an increase in loss while <strong>training</strong>. Or you have trained a model already and when testing, turning shuffle off caused an increase in <strong>test loss</strong>?</p>

username: alx
cooked: <p>Yes because shuffling allows your model to learn about all classes within a batch.</p>
<p>For example, if shuffle = False your first initial batches could only be composed of one class:<br>
0,0,0,0,0,0,0,0,0,0,0,0,0…</p>
<p>In this scenario your model is learning how to predict class 0 very well until it hits the first non-0 class. By then, the loss is massive.</p>
<p>When shuffle = True, your classes are randomly distributed:<br>
0,2,4,1,2,4,0,0,3,1,2,3,5…</p>
<p>So your model learns a little bit about everything. There’s no class it never seen before.</p>

username: Michael-J98
cooked: <p>Yes, it happened during training. Actually, the test set here is validation set. Loss is computed over one batch.</p>

username: Michael-J98
cooked: <p>Thank you for your reply.<br>
But… sorry, I didn’t make myself clear enough before that I was doing a segmention job so the model would see all the classes anyway.</p>

username: Nikronic
cooked: <p>The reason <a class="mention" href="/u/alx">@alx</a> mentioned is the main reason and applies in your case too.<br>
When you are providing your images to the model, if there is particular similarity between many of instances for instance, just providing indoor images/stree/etc, your model will learn less features to understand the differences even though all classes exist in all images(although I many segmentation tasks, this does not apply). So, at the test time/validation, your model did not learn enough about different type of images.</p>
<p>A more mathematical explanation would be the issues regarding updating gradients. For instance, providing an image of street then jungle, enable model to capture more general features rather than overfitting on particular type of color/construction/texture/… .</p>
<p>In the end, I think shuffling data during training is mandatory unless we care about order of data.</p>
<p>bests</p>

username: Michael-J98
cooked: <p>oh~ there’s something wrong with my expression. <img src="https://discuss.pytorch.org/images/emoji/apple/hot_face.png?v=9" title=":hot_face:" class="emoji" alt=":hot_face:"> I was to consider training and validation both in <strong>Training</strong> or  under ‘<strong>model.train()</strong>’. Just validation recorded no gradient by <code>with torch.no_grad():</code> while training does. I just want to log the losses of training set and validation set by evety epoch.<br>
[Can I do that? Put them all under <code>model.train()</code>, or I have to put validation under <code>model.eval()</code>? Would that be the reason?]<br>
And yeah, I shuffled the training set to train the model. But I think there should be no loss difference  whether I turn the shuffle of validation set on or off. After all, the net doesn’t learn about it if it doesn’t provide the gradient. <img src="https://discuss.pytorch.org/images/emoji/apple/face_with_monocle.png?v=9" title=":face_with_monocle:" class="emoji" alt=":face_with_monocle:"></p>

username: Nikronic
cooked: <p>Ow! In first post I asked about training loss or test loss, now I can get what you mean. Actually, after training for a while you are trying to validate your model so, this can be considered as testing. You have to use same configuration with testing which is using <code>model.eval</code> and <code>torch.no_grad</code> for looping over batches.</p>
<aside class="quote no-group" data-username="Michael-J98" data-post="4" data-topic="82335">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/michael-j98/48/20536_2.png" class="avatar"> Michael-J98:</div>
<blockquote>
<p>Loss is computed over one batch.</p>
</blockquote>
</aside>
<p>In this case, when you turn shuffle off, the batches are not identical to the case where shuffle is on for validation set. What may happen is that in ‘off’ case, the validation batch has simpler examples. To find out that your model is working properly or not, you just need to aggregate loss over all batches in case of shuffled and not shuffled.</p>

username: Michael-J98
cooked: <p>OH~ I tried to put validation under <code>model.eval()</code> and it worked! There was no difference between on/off. Thank you very much! <img src="https://discuss.pytorch.org/images/emoji/apple/heart.png?v=9" title=":heart:" class="emoji" alt=":heart:"><br>
BTW, I also tried to compute the loss over one epoch in case of shuffled and not shuffled with validation not under model.eval(). The loss of off-mode was still higher than that of on-mode. Did that mean my model  wasn’t working properly? What’s the principal of that?</p>

username: Nikronic
cooked: <p>You are welcome!<br>
I do not know definition of your model, but I get you are using <code>BatchNorm</code> and maybe <code>Dropout</code>. This layers act differently during <code>eval</code> and <code>train</code> mode. For instance, if <code>model.train</code> is used, then mean/variance in <code>BatchNorm</code> will be updated even on validation set, for every batch, you will have different parameters for all <code>BatchNorm</code> which is not desirable as we looking to update mean/variance in <code>BatchNorm</code> only in training step not testing or validation. That is, after training, in validaiton step, each time a different <code>BatchNorm</code> is being used which has been updated based on previous batch in validation step. So, overall loss will not be same.</p>
<aside class="quote" data-post="1" data-topic="7146">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/liangstein/48/2047_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/what-does-model-eval-do-for-batchnorm-layer/7146">What does model.eval() do for batchnorm layer?</a> 
  </div>
  <blockquote>
    Hi Everyone, 
When doing predictions using a model trained with batchnorm, we should set the model to evaluation model. I have a question that how does the evaluation model affect barchnorm operation? What does evaluation model really do for batchnorm operations? Does the model ignore batchnorm?
  </blockquote>
</aside>

<aside class="quote" data-post="1" data-topic="19615">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/spnova12/48/3495_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/model-eval-vs-with-torch-no-grad/19615">'model.eval()' vs 'with torch.no_grad()'</a> 
  </div>
  <blockquote>
    When i test my model, do I have to use model.eval() even though I am using 'with torch.no_grad() ?
  </blockquote>
</aside>

<p>bests</p>

username: Michael-J98
cooked: <p>Again, truly appreciate your timely help <img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=9" title=":pray:" class="emoji" alt=":pray:"></p>