Model.eval() gives different results every time

assyl_coop: <p>My model looks like this below:</p>
<pre><code class="lang-auto">class GRU(nn.Module):

    def __init__(self, vocab_size, emb_size, hidden_size, num_classes):
        super(GRU, self).__init__()

        self.encoder = nn.Embedding(vocab_size, emb_size)
        self.drop = nn.Dropout(p=0.8)
        self.gru = nn.GRU(emb_size, hidden_size, dropout=0.8, batch_first=True)
        self.bn = nn.BatchNorm1d(hidden_size)
        self.fc = nn.Linear(hidden_size, num_classes)

    def forward(self, x):
        x = self.encoder(x)
        x = self.drop(x)
        x, _ = self.gru(x)
        scores = x.matmul(x.transpose(1, 2))
        scores = F.softmax(scores, dim=1)
        x = scores.matmul(x).sum(1)
        x = self.bn(x)
        x = self.fc(x)
        return x
</code></pre>
<p>I have loaded the model</p>
<pre><code class="lang-auto">model = GRU(vocab_size, embedded_size, hidden_size, num_classes)
model.load_state_dict(torch.load(PATH))
model = model.to(device)
model.eval()
</code></pre>
<p>Also there is my accuracy test function.</p>
<pre><code class="lang-auto">def compute_accuracy(model, data_loader):
    correct_pred, num_examples = 0, 0
    for i, (features, targets) in enumerate(data_loader):

        features = features.to(device)
        targets = targets.to(device)

        logits = model.eval()(features)
        y_pred = logits.max(1)[1]

        num_examples += targets.size(0)
        correct_pred += (y_pred == targets).sum()
    return correct_pred.float() / num_examples * 100
</code></pre>
<p>print(compute_accuracy(model.eval(), valid_loader))<br>
But every time I test model in my dataset, results are different.<br>
am I doing something wrong or .eval() mode not reliable?</p>

ptrblck: <p>This shouldn’t be the case and I can’t see anything obviously wrong.<br>
One side note: could you call <code>model.eval()</code> once at the beginning of your <code>compute_accuracy</code> method and just pass the <code>features</code> as <code>logits = model(features)</code>.</p>
<p>Also, if it’s possible, could you share the <code>state_dict</code> as well as the values for the hyperparameters of your model?<br>
I just tried random values and got the same output after setting <code>model.eval()</code>.</p>

assyl_coop: <p>So my hyperparams are:</p>
<pre><code class="lang-auto">vocab_size = 33988
embedded_size = 500
hidden_size = 300
num_classes = 363
</code></pre>
<p>Modified my compute_accuracy, Results are still different each time.</p>
<pre><code class="lang-auto">def compute_accuracy(model, data_loader, train=False, validation=False):
    correct_pred, num_examples = 0, 0
    model.eval()
    for i, (features, targets) in enumerate(data_loader):

        features = features.to(device)
        targets = targets.to(device)

        logits = model(features)
        y_pred = logits.max(1)[1]

        num_examples += targets.size(0)
        correct_pred += (y_pred == targets).sum()
    return correct_pred.float() / num_examples * 100
</code></pre>
<p><code>state_dict</code> of the model is too big to post here</p>

ptrblck: <p>Thanks for the values!<br>
I couldn’t reproduce this issue using some random inputs:</p>
<pre><code class="lang-python">x = torch.empty(1000, 10, dtype=torch.long).random_(vocab_size)
y = torch.empty(1000, dtype=torch.long).random_(num_classes)
dataset = TensorDataset(x, y)
loader = DataLoader(
    dataset,
    batch_size=1,
    num_workers=1,
    shuffle=False
)

model.eval()

correct1 = 0.0
nb_samples = 0
with torch.no_grad():
    for data, target in loader:
        output = model(data)
        pred = output.argmax(1)
        correct1 += (pred==target).float().sum()
        nb_samples += data.size(0)
acc1 = correct1 / nb_samples

correct2 = 0.0
nb_samples = 0
with torch.no_grad():
    for data, target in loader:
        output = model(data)
        pred = output.argmax(1)
        correct2 += (pred==target).float().sum()
        nb_samples += data.size(0)
acc2 = correct2 / nb_samples

print(correct1, acc1)
print(correct2, acc2)
</code></pre>
<p>I noticed one possible issue: Could you cast your comparison to float before summing it:</p>
<pre><code class="lang-python">correct_pred += (y_pred == targets).float().sum()
</code></pre>
<p>Depending on the batch size you are using, you might encounter an overflow here.</p>

assyl_coop: <pre><code class="lang-auto">def compute_accuracy(model, data_loader, train=False, validation=False):
    correct_pred, num_examples = 0, 0
    model.eval()
    for i, (features, targets) in enumerate(data_loader):

        features = features.to(device)
        targets = targets.to(device)

        logits = model(features)
        y_pred = logits.max(1)[1]

        num_examples += targets.size(0)
        correct_pred += (y_pred == targets).float().sum()
    return correct_pred.float() / num_examples * 100
</code></pre>
<p>results are:<br>
first:</p>
<pre><code class="lang-auto">tensor(81.4359, device='cuda:0')
tensor(75.7375, device='cuda:0'
</code></pre>
<p>second time:</p>
<pre><code class="lang-auto">tensor(81.4284, device='cuda:0')
tensor(75.7506, device='cuda:0')
</code></pre>

ptrblck: <p>Are these numbers you are printing <code>correct_pred</code> and <code>(correct_pred / num_examples * 100)</code>?</p>

assyl_coop: <aside class="quote no-group" data-username="assyl_coop" data-post="6" data-topic="34865">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/e495f1/48.png" class="avatar"> assyl_coop:</div>
<blockquote>
<p>correct_pred.float() / num_examples * 100</p>
</blockquote>
</aside>
<pre><code class="lang-auto">correct_pred.float() / num_examples * 100
</code></pre>
<p>Yes</p>

ptrblck: <p>Why is <code>correct_pred</code> not an integer (not the <code>dtype</code> but the value), if you just sum <code>(y_pred == targets)</code>?</p>

assyl_coop: <pre><code class="lang-auto">return correct_pred / num_examples * 100
</code></pre>
<p>tried, still results are diff.</p>
<pre><code class="lang-auto">tensor(81.4359, device='cuda:0')
tensor(75.7331, device='cuda:0')
</code></pre>
<p>and</p>
<pre><code class="lang-auto">tensor(81.4478, device='cuda:0')
tensor(75.7375, device='cuda:0'
</code></pre>

rasbt: <p>For debugging purposes, maybe also try</p>
<pre><code>(y_pred.long() == targets.long()).sum()
</code></pre>
<p>(but I guess if they were not longs before you would get an error).</p>
<p>Another thing is, I am not exactly sure what your training loader does. Usually, shuffling wouldn’t affect the accuracy because you iterate through all examples anyway and don’t update the weights. However, since you have some hidden state, maybe there’s some shuffling in the dataset that may affect it (if you are constructing sentences from texts, for example). So maybe try to turn the shuffling off in the dataset loader if you haven’t done so and see if that has some effect.</p>

assyl_coop: <p>Thnak you!<br>
It worked. I just changed shuffle to False.</p>

saikrishnarallabandi: <p><a class="mention" href="/u/rasbt">@rasbt</a></p>
<p>sorry I dont get it. can you please expand?</p>
<p>the GRU hidden will not get updated during eval mode correct? why would shuffling make a difference in that case?</p>

rasbt: <aside class="quote no-group" data-username="saikrishnarallabandi" data-post="13" data-topic="34865">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/saikrishnarallabandi/48/22496_2.png" class="avatar"> saikrishnarallabandi:</div>
<blockquote>
<p>the GRU hidden will not get updated during eval mode correct? why would shuffling make a difference in that case?</p>
</blockquote>
</aside>
<p>Yeah, but it is still sequence data. E.g., a text with a random sentence order would not make so much sense as having he sentences in the correct order.</p>

Joe_Renner: <p>Hi,</p>
<p>I’m coming in a bit late here, but I’m running into a problem related to this. I am using GRU layers for a sequence classification problem where I take the last hidden state as input to dense layers.<br>
I get different outputs for the same data and same model weights if I either set the model to .eval() or shuffle the data. I was under the impression the GRU hidden state is set to zero for all sequences in a batch (regardless if in eval or not), so why would shuffling the data matter? In other words, is it not the case that every example in a batch is independent?</p>
<p>Thanks</p>

Brinda: <aside class="quote no-group" data-username="assyl_coop" data-post="1" data-topic="34865">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/e495f1/48.png" class="avatar"> assyl_coop:</div>
<blockquote>
<p><code>data_loader</code></p>
</blockquote>
</aside>
<p>Hi. I am facing the exact same issue of getting different accuracy results from model.eval(). Would you mind sharing where you turned off Shuffle argument? I have 2 dataloaders: train_loader which has batches of training data and test_loader which has batches of testing data.</p>