[DataLoader Problem] Problem arises when shuffle = True

basingse: <p>I have training , validation and test dataset(NLP problem , So I used LSTM , GRU) . The model contains batch norm layer (I think this is the reason for discrepancy I am observing). I don’t have true labels for test dataset. This was my training procedure before :</p>
<ul>
<li>Train on training dataset for 5 epochs (model.train() was used). For each epoch after the training I make prediction on validation dataset to see my results(model.eval() was used) and then save the model if the validation score(AUC score as metric) increased. At last I load the best model and then make predictions on test data (model.eval() was used).<br>
This is my pipeline:</li>
</ul>
<pre><code class="lang-auto">for epoch_number in range(epochs):
       Train the model on training dataset.
       Make predictions on validation
       save the model using torch.save(model.state_dict(),fname) if the validation score increased.

Load the best model and then predictions on test data.
</code></pre>
<p>For 5 epochs my train loss and validation loss are</p>
<pre><code class="lang-auto">1) train loss =  0.2599791 , val loss =  0.2254444
2) train loss =  0.2198705 , val loss =  0.2254712
3) train loss =  0.2080045 , val loss =  0.2124491
4) train loss =  0.1860864 , val loss =  0.18708
5) train loss =  0.1701995 , val loss =  0.1935813
</code></pre>
<p>Recently I changed the procedure a little bit. It is as follows :</p>
<ul>
<li>Train on training dataset for 5 epochs(model.train() was used), For each epoch after the training I make predictions on both validation and test dataset(model.eval() was used for both) and then save the validation and test predictions for future use. This is my pipeline</li>
</ul>
<pre><code class="lang-auto">for epoch_number in range(epochs):
       Train the model.
       Make predictions on validation and test.
       save the predictions.
</code></pre>
<p>This is where I am seeing some discrepancy. As it’s obvious that train loss and validation loss must remain same because I am making predictions  at every epoch and this will not update the weights. But these are my train loss and validation loss</p>
<pre><code class="lang-auto">1) train loss =  0.2599791 , val loss =  0.2254444
2) train loss =  0.2196528 , val loss =  0.2283426
3) train loss =  0.2078255 , val loss =  0.1996013
4) train loss =  0.1848111 , val loss =  0.182577
5) train loss =  0.1680537 , val loss =  0.1896651
</code></pre>
<p>Note that the first epoch train loss and validation loss in both cases is same (This the sole reason I have provided these numbers) and then there are changes after 1st epoch.</p>
<p>Why this discrepancy is happening?</p>
<p>I have been thinking of this problem for some days and here are my thoughts on this</p>
<ul>
<li>When you train with batch norm it keeps running estimates and then use these running estimates in model.eval() mode. But I think in the eval mode the running estimates are also changed and this is the reason for discrepancy. May be I am wrong it’s just my view.</li>
</ul>

amosella: <p>The are different factors than can produce this behavior. First at all, Are you seeding all frameworks that you are using to generate random numbers? Are you setting cudnn to determinisic mode? You can do it using this piece of code:</p>
<blockquote>
<p>torch.backends.cudnn.deterministic = True<br>
torch.backends.cudnn.benchmark = False</p>
</blockquote>
<p>Moreover you need to take into account some gpu operations are non-deterministic This <a href="https://pytorch.org/docs/stable/notes/randomness.html" rel="noopener nofollow ugc">link</a> can be useful for you. (I am supposing you are using GPU.)</p>
<p>Answering your last point. As far as I know batchnorm is not updated in evaluation time. You can find more information about batchnorm <a href="https://arxiv.org/pdf/1502.03167v3.pdf" rel="noopener nofollow ugc">here</a> .</p>

basingse: <pre><code class="lang-auto">def seed_everything(SEED=42):
    random.seed(SEED)
    np.random.seed(SEED)
    torch.manual_seed(SEED)
    torch.cuda.manual_seed(SEED)
    torch.cuda.manual_seed_all(SEED)
    torch.backends.cudnn.deterministic = True
    os.environ['PYTHONHASHSEED']=str(SEED)
    # torch.backends.cudnn.benchmark = False

SEED=42
seed_everything(SEED=SEED)
</code></pre>
<p>I am seeding everything as shown above. Each part is reproducible but when I change from one part to other one , I am observing this. Can I know whats the problem ?</p>

amosella: <p>Do you mean that if you execute X times the first pipeline you are obtaining the same result? I suppose that you did not change the order of init of your model, dataloaders, right? Without more information about your network and your code it is difficult to me to know what is the problem. If you provide a piece of code to reproduce this behavior I can take a look.</p>
<p>Moreover,  I think it will be a good idea to set <code>cudnn.benchmark</code> to false if your input sizes changes at each iteration. <code>cudnn.benchmark</code> choose the best algorithm for your input data. That can be a source of randomness. It is quite possible the cudnn lstm kernels are not deterministic as it is said <a href="https://discuss.pytorch.org/t/non-deterministic-result-on-multi-layer-lstm-with-dropout/9700">here</a>.</p>

basingse: <p>I am sorry for not giving the enough details. The first and second pipeline are reproducible. (I have tried running for 3 to 4 times and results are the same). I checked twice and I am not changing the dataloader for training , validation and test dataset.</p>
<p>Let’s suppose there is some problem with <code>cudnn</code> , then how come the first epoch losses are same in the both the pipelines.</p>
<p>I also tried this: When I don’t make predictions on test data in second pipeline then the losses and AUC score on validation data are same as first pipeline.</p>

amosella: <p>I see your point. One last question, Are you shuffling data in validation and test sets? If the answer is yes, this can be the problem. As I said previously without more information about your network, and training code it is not possible to me to guess what is the problem.</p>

basingse: <p>I forgot to mention this point. Shuffling is done only on train and not on validation and test.</p>
<p>The code that I have written is for a kaggle competition , so I can’t share the code totally.</p>
<p>Some details are (Some things I have mentioned above but writing<br>
model : I am using LSTM’s and GRU’s in my model.<br>
Seed : Everything is seeded as I have mentioned above.<br>
Reproducible : Both pipelines are totally reproducible. Only discrepancy is shift from piepliene 1 to pipeline 2.</p>

amosella: <p>I have the impression that something is shifting the random state in your pipeline, but I can not be sure, you can check this deactivating the shuffle in trainning dataset (in both pipelines). I do not have more ideas at this moment, I am sorry <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"> Maybe someone else can give a hint to you.</p>

basingse: <p>amosella, you are right I think there is something that is shifting the random state in my 2nd pipeline. So when I kept shuffle = False for training dataset both pipelines produced the same result.</p>
<p>Other things I tried are:</p>
<ul>
<li>Removed batch norm from the model and shuffle = True gave different results(from 2nd epoch onwards ) implies there is no problem with batch norm (I have the changed title name of the thread).</li>
</ul>
<p>As 1st epoch results are same for both the pipelines I think adding test predictions is changing the random state.</p>
<ul>
<li>removed test predictions from the second pipeline and added validation predictions one more time. It is shown below</li>
</ul>
<pre><code class="lang-auto">for epoch_number in range(epochs):
        Train the model
        Make predictions on validation 
        Make predictions on validation             # 2nd time
        save the model if validation score increases..
</code></pre>
<p>and to my surprise the results are same as my second pipeline.</p>

amosella: <p>Hmm, are you doing data augmentation on validation and test datasets? Somewhere you are calling a random method that produce this shifting in the random state.</p>

basingse: <p>No augmentation on any of the datasets.</p>
<aside class="quote no-group" data-username="amosella" data-post="10" data-topic="45631">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/amosella/48/12205_2.png" class="avatar"> amosella:</div>
<blockquote>
<p>Somewhere you are calling a random method that produce this shifting in the random state.</p>
</blockquote>
</aside>
<p>I will go through once more</p>

basingse: <p>amosella, this is my code structure , and I couldn’t figure where is the problem <img src="https://discuss.pytorch.org/images/emoji/apple/frowning_face.png?v=9" title=":frowning_face:" class="emoji" alt=":frowning_face:"></p>
<pre><code class="lang-auto">def  fit_data():
      for epoch_num in range(epochs):
         for train_data in train_iterator:
             predict
             calculate loss
             backprop
          
          for val_data in val_iterator:
             predict
          
          for test_data in test_iterator:
             predict
      save the validation and test predictions.
</code></pre>
<p>the iterators for all the datasets are output of the corresponding dataloaders . I don’t know where it is inducing the shift in random state.</p>

amosella: <p>Can you put shuffle=false in the data loader? in the documentation it is said that shuffle is default false. Can you verify it? Moreover put workers to 0 and check it.</p>

basingse: <aside class="quote no-group" data-username="basingse" data-post="9" data-topic="45631">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/839c29/48.png" class="avatar"> basingse:</div>
<blockquote>
<p>when I kept shuffle = False for training dataset both pipelines produced the same result.</p>
</blockquote>
</aside>
<p>I put workers = 0 and results are different.</p>

amosella: <p>I do not have more ideas <img src="https://discuss.pytorch.org/images/emoji/apple/cry.png?v=9" title=":cry:" class="emoji" alt=":cry:">. I think that you can use this <a href="https://pytorch.org/docs/stable/torch.html?highlight=get_rng#torch.get_rng_state" rel="nofollow noopener">function</a> to look who is changing the state (using prints… )</p>

basingse: <p>Thanks for the help amosella. <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"></p>

basingse: <p>So I could reproduce my problem. The link to code is given below :</p>
<ul>
<li>
<p>train + validation :  <a href="https://www.kaggle.com/suchith0312/pytorch-dataloader-testing?scriptVersionId=14425205" rel="nofollow noopener">https://www.kaggle.com/suchith0312/pytorch-dataloader-testing?scriptVersionId=14425205</a></p>
</li>
<li>
<p>train + validation + test : <a href="https://www.kaggle.com/suchith0312/pytorch-dataloader-testing?scriptVersionId=14425182" rel="nofollow noopener">https://www.kaggle.com/suchith0312/pytorch-dataloader-testing?scriptVersionId=14425182</a></p>
</li>
</ul>
<p>As you can see from the output that train_id is different for two cases which must not happen. My original code is of same format.</p>

InnovArul: <aside class="quote no-group" data-username="basingse" data-post="17" data-topic="45631">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/839c29/48.png" class="avatar"> basingse:</div>
<blockquote>
<p>As you can see from the output that train_id is different for two cases which must not happen. My original code is of same format.</p>
</blockquote>
</aside>
<p>Maybe, I am a bit late to the party. Why do you think <code>train_id</code>s should not be different?</p>

basingse: <aside class="quote no-group" data-username="InnovArul" data-post="18" data-topic="45631">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/innovarul/48/5282_2.png" class="avatar"> InnovArul:</div>
<blockquote>
<p>Maybe, I am a bit late to the party. Why do you think <code>train_id</code> s should not be different?</p>
</blockquote>
</aside>
<p>I am iterating over train only once in an epoch in both the codes. So I expect that the id’s in both cases must be same or I am missing something</p>

InnovArul: <p>I think there is a misunderstanding. I see that train ids in both codes are same.</p>
<p>First script:</p>
<pre><code class="lang-auto">15 examples of train
tensor([76., 59., 48., 42., 32., 49., 70.,  3., 98., 63., 90., 25., 91., 92.,
        51.])

</code></pre>
<p>Second script:</p>
<pre><code class="lang-auto">15 examples of train
tensor([76., 59., 48., 42., 32., 49., 70.,  3., 98., 63., 90., 25., 91., 92.,
        51.])
</code></pre>
<p>Or did I misunderstand your question?</p>