Using DataLoader yields different results for shuffle: (True/False)

fmsilver: <p><strong>Problem:</strong><br>
I have a testset of samples that is too large for classification in one single run (memory error).<br>
The testset is structured as: [0…1…2] where there is 400 ‘0’, 400 ‘1’ and 400 ‘2’ =&gt; 1200 samples.<br>
(The trained model yields ~80% validation accuracy =&gt; I expect ~80% in test accuracy.)</p>
<p><strong>Implemented solution:</strong><br>
<code>test_loader = DataLoader(dataset=testset,batch_size=400,shuffle=False)</code><br>
Result: test accuracy of batches: [25%,65%,35%] =&gt; Can not be correct!</p>
<p>If I change to:<br>
<code>test_loader = DataLoader(dataset=testset,batch_size=400,shuffle=True)</code><br>
Result: test accuracy of batches: [77%,80%,79%] =&gt; This seems legit!</p>
<p>How can the results be so different if the dataloader load batches of mixed classes instead of batches with only one class? I am baffled, the model should not care about what it classifies!?</p>
<p><strong>Code:</strong></p>
<blockquote>
<pre><code>testset        = DATA(train_X,train_Y)
test_loader    = DataLoader(dataset=testset,batch_size=400,shuffle=False)
for i, data in enumerate(test_loader, 0):
    x_test, y_test = data
    with torch.no_grad():
        output_test = model(x_test.cuda().float())
    preds_test      = np.argmax(list(torch.exp(output_test).cpu().numpy()), axis=1)
    acc_test        = accuracy_score(y_test, preds_test)
    print(acc_test)
</code></pre>
</blockquote>

fmsilver: <p>I have no idea as to why the above occurs, but I was able to go around the “mystery” by the following, for future readers:</p>
<p><strong>1.</strong> Return the indices of the batches from the class of your Dataset</p>
<pre><code>class DATA(Dataset):

    def __init__(self,x,y,transform):
        self.x_data    = x
        self.y_data    = y
        self.len = self.x_data.shape[0]

    def __getitem__(self, index):
        X_data = self.x_data[index]
        Y_data = self.y_data[index]
        return X_data,Y_data, index  ########## Returns the indices

    def __len__(self):
        return self.len
</code></pre>
<p><strong>2.</strong> Insert the prediction at the correct indices:</p>
<blockquote>
<p>preds = np.array(train_Y)          ## Copies the whole target vector (1200 samples in my case)</p>
</blockquote>
<blockquote>
<p>for i, data in enumerate(test_loader, 0):</p>
</blockquote>
<blockquote>
<pre><code>x_test, y_test, index = data
index = index.detach().numpy()
with torch.no_grad():
    output_test = model(x_test.cuda().float())
preds_test      = np.argmax(list(torch.exp(output_test).cpu().numpy()), axis=1)
preds[index]    = preds_test   ### Insert batch (400 samples) predictions at correct indices in the 1200 array.
</code></pre>
</blockquote>

ptrblck: <p>How are you calculating the accuracy given your predictions?<br>
Are you setting your model to <code>.eval()</code> before executing the test pass?</p>

fmsilver: <p>Accuracy score by:<br>
‘from sklearn.metrics import accuracy_score’</p>
<p>And the model is the last one after training has ended, it is not loaded and I do not call ‘model.eval()’<br>
It works with the indices, so you really do not have to spend time on this topic, but big thanks for being here <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p>Good to hear it’s working, but I would still make sure there is no bug in PyTorch somewhere. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
<p>If you keep the model in <code>model.train()</code> and use e.g. batch norm layers, the running statistics will still be updated.<br>
If you shuffle the test data, these stats should represent the dataset better than a non-shuffled dataset.<br>
Note that this could be considered a data leakage.</p>

fmsilver: <p>Changed to model.eval() before prediction.<br>
And it is now producing the same results for both shuffle: ‘False’ and ‘True’.<br>
(I have both Dropouts and batch norm layers in my model)</p>
<p>Thanks! And no bug in Pytorch!</p>

janmejaya: <p>hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,<br>
Can you please explain a bit more about how the batch norm stats causing data leakage(when we shuffle the data on the validation set)?<br>
My current model’s (validation)results have 20-25% difference if I change this single parameter(ie. <code>shuffle=True</code> to <code>shuffle-False</code>). Ideally, I shouldn’t shuffle the data on validation set but these results are quite deceptive.</p>

ptrblck: <p>If you don’t switch to <code>.eval()</code> before using the evaluation dataset (or test set), the batch norm stats will be updated, so your model will “train” using this dataset, which is considered a data leakage.<br>
Shuffling the validation or test dataset might even result in “better” stats to leak.</p>
<p>Note that shuffling does not change anything, if you set the model properly to <code>eval()</code> before validating the model.</p>

janmejaya: <p>Got your point regarding <code>.eval()</code> mode, but I fear that shuffle is changing my validation results. My code snippet looks something like this</p>
<pre><code class="lang-auto">for each epoch:
    self.model.train()
    # train data for each batch
    self.model.eval()
    # validate
</code></pre>
<p>my validation results differ by 20-25% with the change of the <code>shuffle</code> parameter. Note that I am changing the <code>shuffle</code> parameter of my validation data loader (not train data loader).</p>
<p>While debugging I stumbled upon <a href="https://stackoverflow.com/questions/58502316/pytorch-model-train-and-model-eval-behave-in-a-weird-way" rel="nofollow noopener">this StackOverflow</a> question and surprisingly OP’s 2nd scenario prefectly fits mine. When i turn off the <code>shuffle</code> (on my validation data loader) there is a huge difference between my train and validation accuracy(train being 95% while validation stuck on 71%) , but the difference vanishes as the <code>shuffle</code> turned on.</p>

ptrblck: <p>What model are you using?<br>
Do you have a custom model with e.g. functional dropout calls in your <code>forward</code> method?<br>
Could you post a code snippet to reproduce this issue?</p>

ryuxin: <p>I have the same problem as well. My model is fc-&gt;bn-&gt;relu-&gt;fc-&gt;bn-&gt;logsoftmax<br>
I have already set model.eval() before test,</p>
<p>If I set (shuffle=true), the accuracy is about Accuracy: 2270/3200 (70.94%)<br>
Testing accuracy: 70.94</p>
<p>If I set (shuffle=false), the accuracy is about Test set: Average loss: 3.3379<br>
Accuracy: 757/3200 (23.66%)<br>
Testing accuracy: 23.66</p>
<p>There is a huge difference</p>

ptrblck: <p>Could you post your code so that we could have a look?</p>

lucab: <p>Hi everybody,<br>
I have a similar problem. I have trained my model (a CNN with batch normalizations and dropouts) and stopped the learning process after I reached ~80% on the validation set (using dataloader with suffle=True). I saved my model with</p>
<pre><code class="lang-auto">torch.save(self.model.state_dict(), filename)
</code></pre>
<p>I wrote another script, with the following code:</p>
<pre><code class="lang-auto">def simple_validator(device, model_file, data_folder, chunk_size):
    val_data = riq.IQDataset(data_folder=data_folder, chunk_size=chunk_size, validation=True)
    val_data.normalize(torch.tensor([-3.1851e-06, -7.1862e-07]), torch.tensor([0.0002, 0.0002]))
            
    model = brain.CharmBrain(chunk_size)
    model.load_state_dict(torch.load(model_file))
    model.to(device)
    model.eval()
            
    tot = 0 
    correct = 0
    with torch.no_grad():
        for chunk, label in val_data: 
            chunk = chunk.to(device, non_blocking=True)
            output = model(chunk.unsqueeze(0))
            _, predicted = torch.max(output, dim=1) 
            if predicted.item() == label:
                correct += 1
            tot += 1
    print(f"Accuracy: {correct/tot}")
                    
                        
def validator(device, model_file, data_folder, chunk_size):
    val_data = riq.IQDataset(data_folder=data_folder, chunk_size=chunk_size, validation=True)
    val_data.normalize(torch.tensor([-3.1851e-06, -7.1862e-07]), torch.tensor([0.0002, 0.0002]))
    val_loader = torch.utils.data.DataLoader(val_data, batch_size=512, shuffle=True, num_workers=20, pin_memory=True)
                
    model = brain.CharmBrain(chunk_size) 
    model.load_state_dict(torch.load(model_file))
    model.to(device)
    model.eval()
    
    tot = 0
    correct = 0
    with torch.no_grad():
        for chunks, labels in val_loader:
            chunks = chunks.to(device, non_blocking=True)
            labels = labels.to(device, non_blocking=True)
            output = model(chunks)
            _, predicted = torch.max(output, dim=1)
            correct += int((predicted == labels).sum())
            tot += labels.shape[0]
        print(f"Accuracy: {correct/tot}")
     
</code></pre>
<p>The two functions (validator and simple_validator) operate with the same model parameter file and the same data; however:</p>
<ol>
<li>multiple runs of simple_validator return the <strong>same value</strong>,  ~0.336</li>
<li>multiple runs of validator return <strong>different values</strong>, but all around ~0.81</li>
<li>multiple runs of validator setting <strong>shuffle=False</strong> return <strong>different values</strong>, but all around 0.3365</li>
</ol>
<p>(Note, this model is a classifier with 3 classes, so accuracy of 0.33 means random output)</p>
<p>Any idea, where the bug could be?</p>
<p>Thanks,<br>
Luca</p>

lucab: <p>I retrained my model without Batch normalization and AlphaDropout. During learning, I could achieve an accuracy on the validation set (through the DataLoader, shuffle=True) of ~0.48, and now:</p>
<ol>
<li>validator obtains different values, but in the same range of ~0.48 w/ and w/o shuffle=True</li>
<li>the result of validator and simple validator always coincide to the same value when num_workers=1 (with both shuffle=True, and shuffle=False)</li>
</ol>
<p>My naive conclusions: model.eval() does not completely fix the values of the batch normalization and dropout layers; there is something weird happening when using multiple processes to load the data.</p>
<p>Can anybody explain any of these results?</p>

ptrblck: <p><code>model.eval()</code> should use the running stats, if the batchnorm layers were created in the default setup. Could you check, if you are initializing them with <code>track_running_stats = False</code>?<br>
Also, dropout should be disabled, if the <code>nn.Dropout</code> module is used or the functional API via <code>F.dropout(input, training=self.training)</code>. Could you check, if you are using the functional API and are not switching the <code>training</code> flag?</p>

lucab: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, thank you for your reply; indeed I was using <code>track_running_stats=False</code>. However, I am using dropouts with default settings (even if they are AlphaDropouts, to be precise).<br>
If in validation, batch norm uses the wrong stats, that could explain the drop in performance.<br>
However, I am more puzzled by why using a loader (num_workers=1) gives different results wrt to not use it; and, by why using multiple loading processes (num_workers&gt;1) gives different values at each run.</p>

ptrblck: <aside class="quote no-group" data-username="lucab" data-post="16" data-topic="59375">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lucab/48/37679_2.png" class="avatar"> lucab:</div>
<blockquote>
<p>If in validation, batch norm uses the wrong stats, that could explain the drop in performance.</p>
</blockquote>
</aside>
<p>I wouldn’t say it’s using the wrong stats, but the batch statistics will be used, if <code>track_running_stats=True</code> is set.</p>
<aside class="quote no-group" data-username="lucab" data-post="16" data-topic="59375">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lucab/48/37679_2.png" class="avatar"> lucab:</div>
<blockquote>
<p>However, I am more puzzled by why using a loader (num_workers=1) gives different results wrt to not use it; and, by why using multiple loading processes (num_workers&gt;1) gives different values at each run.</p>
</blockquote>
</aside>
<p>If you are using 3rd party libs in the data loading pipeline, you might need to seed them in the <code>worker_init_fn</code>. Since the <code>fork</code> method is used for multiprocessing on Linux, the random states might be cloned from the parent process.</p>

lucab: <p>Hi,<br>
it turned out the dataset class I had been using had a race condition bug. Whenever I used more than<br>
one process to fetch the data, I stumbled with weird data.<br>
Once fixed the bug and removed the track_running_stats=False flag, everything started to work as expected.</p>
<p>Best,<br>
Luca</p>

San_Working: <p>Running into a similar crazy dumb problem.<br>
<code>model.eval()</code> before testing help <img src="https://discuss.pytorch.org/images/emoji/apple/laughing.png?v=12" title=":laughing:" class="emoji" alt=":laughing:" loading="lazy" width="20" height="20"></p>