Shuffling DataLoader causing AucRoc falls down

username: Jona
cooked: <p>Hi,</p>
<p>I’m playing with de Adults Dataset (UCI Repository), and I’m experiencing something I don’t have an explanation about it</p>
<p>This is my model</p>
<pre><code class="lang-auto">class UciAdultsClassifier(nn.Module):
    def __init__(self, q_continius_features:int, q_categorical_features:int, embedding_dims:list):
        super(UciAdultsClassifier, self).__init__()
        
        embedding_sizes = sum([embedding_size for _, embedding_size in embedding_dims])
        
        self.embeddings_layer=nn.ModuleList(
            [nn.Embedding(vocabulary_size, embedding_size) for vocabulary_size, embedding_size in embedding_dims]
        )
        
        self.embedding_dropout = nn.Dropout(0.6)
        
        self.layer1=nn.Sequential(
            nn.Linear(embedding_sizes + q_continius_features, 128),
            nn.ReLU(),
            nn.BatchNorm1d(128),            
        )
    
        self.layer2=nn.Sequential(
            nn.Linear(128, 64),
            nn.ReLU(),
            nn.Dropout(0.3),
            nn.BatchNorm1d(64),            
        )
        
        self.layer3=nn.Sequential(
            nn.Linear(64, 32),
            nn.ReLU(),
            nn.Dropout(0.3),
            nn.BatchNorm1d(32),            
        )
        
        self.output=nn.Sequential(
            nn.Linear(32, 1),
            nn.ReLU(),
            nn.Dropout(0.3),
            nn.BatchNorm1d(1),
            nn.Sigmoid()
        )
        
    def forward(self, continius_features, categorical_features):
        embeds = [emb_layer(categorical_features[:, i]) for i, emb_layer in enumerate(self.embeddings_layer)] 
        embeds = torch.cat(embeds, 1)
        
        x = self.embedding_dropout(embeds)
        x = torch.cat([embeds, continius_features], 1)
        x = self.layer1(torch.cat([embeds, continius_features], 1))
        x = self.layer2(x)
        x = self.layer3(x)
        
        return self.output(x)
    
    def fit(self, train_dl:DataLoader, epochs:int, opt:Optimizer, loss_fn:any) -&gt; list:
        self.train()
        losses = []
        for i in range(epochs):
            for x_continius, x_categorical, y in train_dl:
                y_pred = self.forward(x_continius, x_categorical)
                loss = loss_fn(y_pred, y)
                losses.append(loss.item())

                loss.backward()
                opt.step()
                opt.zero_grad()
        return losses
    
    def predict(self, data_loader:DataLoader) -&gt; torch.Tensor:
        self.eval()
        predictions = []
        with torch.no_grad():
            for x_continius, x_categorical, y in data_loader:
                preds = self.forward(x_continius, x_categorical)
                predictions.append(preds)
        return torch.cat(predictions)
</code></pre>
<p>My Dataloaders</p>
<pre><code class="lang-auto">train_dl = DataLoader(train_ds, batch_size=1000, shuffle=False)
test_dl = DataLoader(test_ds, batch_size=1000, shuffle=False)
</code></pre>
<p>My settings:</p>
<pre><code class="lang-auto">model = UciAdultsClassifier(q_continius_features=q_continius_columns, q_categorical_features=q_categorical_columns, embedding_dims=embedding_dims)
optimizer = optim.Adam(model.parameters(), lr=1e-2)
bceloss_fn = nn.BCELoss(reduction='mean')
epochs=100

losses = model.fit(train_dl=train_dl, epochs=epochs, loss_fn=bceloss_fn, opt=optimizer)
</code></pre>
<p>My metrics</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/4/a40f84b12987e68ac3d20933fc0a5a71a834e18e.png" data-download-href="https://discuss.pytorch.org/uploads/default/a40f84b12987e68ac3d20933fc0a5a71a834e18e" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/4/a40f84b12987e68ac3d20933fc0a5a71a834e18e_2_690x363.png" alt="image" data-base62-sha1="nplyhNPnqPXPW4uTDlClQMzSFc2" width="690" height="363" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/4/a40f84b12987e68ac3d20933fc0a5a71a834e18e_2_690x363.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/4/a40f84b12987e68ac3d20933fc0a5a71a834e18e_2_1035x544.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/4/a40f84b12987e68ac3d20933fc0a5a71a834e18e_2_1380x726.png 2x" data-dominant-color="F1F0F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1564×824 114 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>When I change the Dataloader to this line (shuffle=True)</p>
<pre><code class="lang-auto">train_dl = DataLoader(train_ds, batch_size=1000, shuffle=True)
test_dl = DataLoader(test_ds, batch_size=1000, shuffle=False)
</code></pre>
<p>My training Auc Roc falls down to 50%</p>
<p>Why is this happening? Let me know if you want to check de NB.</p>
<p>Best regards</p>

username: ptrblck
cooked: <p>I’m not sure in which situations shuffling might hurt the performance, but a possibility (even though I don’t think it’s highly likely) would be that the non-shuffled training dataset represents the batch statistics better towards the end of the training data and thus the test dataset would see a benefit in better adapted running stats in the batchnorm layers.<br>
This effect can sometimes be observed in time-dependent data, where e.g. a sensor captured the signal and the first part of the data is used as training while the latter would be used as the validation dataset.</p>
<p>Could this be the case for your dataset or are all samples independently sampled?</p>

username: Jona
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> …</p>
<p>I finally found de answer. When I create my training dataloader with <code>shuffle=True</code>, there is no harm in fitting the model.</p>
<p>The problem came after, when using the same shuffled training dataloader for predictions which caused a disaster since the orders were altered, so those predictions were no longer ordered as the <code>y_train</code> expected. In a nutshell, the samples were unordered …</p>
<p>I created a second dalaloder, thsi way:</p>
<pre><code class="lang-auto">train_dl = DataLoader(train_ds, batch_size=1000, shuffle=True)

train_dl_for_prediction = DataLoader(train_ds, batch_size=1000, shuffle=False)
test_dl = DataLoader(test_ds, batch_size=1000, shuffle=False)
</code></pre>
<p>then … I fitted the model…</p>
<p><code>losses = model.fit(train_dl=train_dl, epochs=epochs, loss_fn=bceloss_fn, opt=optimizer)</code></p>
<p>But when calling to <code>predict</code>, I did it this way</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/4/7464ce50157532a809aa398ea6b821efbbfb1cb1.png" data-download-href="https://discuss.pytorch.org/uploads/default/7464ce50157532a809aa398ea6b821efbbfb1cb1" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/7464ce50157532a809aa398ea6b821efbbfb1cb1_2_690x393.png" alt="image" data-base62-sha1="gBFmsreTT1iSOajoV9pL9qpaE9z" width="690" height="393" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/7464ce50157532a809aa398ea6b821efbbfb1cb1_2_690x393.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/7464ce50157532a809aa398ea6b821efbbfb1cb1_2_1035x589.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/7464ce50157532a809aa398ea6b821efbbfb1cb1_2_1380x786.png 2x" data-dominant-color="F1F0F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1568×894 120 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I don’t know if there is a cleaner way to do this, that is, using the original training dataloader, but unshuffled …</p>
<p>Thanks for helping me …</p>
<p>Best regards <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>