Evaluate Twice, Accuracy Changes if I Shuffle

4gatepylon: <p>I have a sanity test for my model when I need to use it for evaluation. I do not understand why the accuracy changes every time I evaluate. It seems to change on some sort of period if I just keep evaluating, so I’m guessing the shuffling order will change periodically in some way (maybe the index the Dataloader is starting at changes or something like that).</p>
<p>My models are ResNet10 on CIFAR-10.</p>
<p>I’ve tried converting the tensors to doubles in the evaluate function (as shown below) to fix numerical issues but to no avail.</p>
<p>I’m fixing all other sources of randomness I know. <strong>Why would this happen? Why would shuffling the data change the accuracy like that?</strong></p>
<pre><code class="lang-auto">def fix_seed(seed):
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed(seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False
</code></pre>
<pre><code class="lang-auto">  acc = evaluate(model, test_loader)
  print("Model 1 Original Accuracy: {}".format(acc))
  assert acc == evaluate(model, test_loader)
</code></pre>
<p>and</p>
<pre><code class="lang-auto">def evaluate(model, test_loader):
    model.eval()

    # TODO, why would the total_correct / total_num change as we shuffled the data differently?
    for _, (images, labels) in enumerate(test_loader):
        total_correct, total_num = 0., 0.

        with torch.no_grad():
            labels = labels.cuda().double()
            img = images.cuda()
            h = model(img)
            preds = h.argmax(dim=1).double()
            total_correct = (preds == labels).sum().cpu().item()
            total_num += h.shape[0]

    return total_correct / total_num
</code></pre>

Andrei_Cristea: <p>Hello! Just a quick note / question. Below:</p>
<aside class="quote no-group quote-modified" data-username="4gatepylon" data-post="1" data-topic="150976">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/4gatepylon/48/48373_2.png" class="avatar"> 4gatepylon:</div>
<blockquote>
<pre><code class="lang-auto">  acc2 = evaluate(model, test_loader)
  print("Model 1 Original Accuracy: {}".format(acc))
  assert acc == evaluate(model, test_loader)
</code></pre>
</blockquote>
</aside>
<p>How do you define <code>acc</code>? This snippet shows where you define <code>acc2</code> but not <code>acc</code>. And you’re comparing <code>acc</code> rather than <code>acc2</code> with the re-evaluated model, so just wanted to confirm that this isn’t somehow causing the issue.</p>
<p>Also, antother quick idea is: have you tried also adding this to your list of deterministic instructions:<br>
<code>torch.use_deterministic_algorithms(True)</code></p>
<p>Per the documentation, that does more than just <code>torch.backends.cudnn.deterministic = True</code> even though they sound similar.</p>

4gatepylon: <p>Ok thanks for the tip with the deterministic algorithms! Regarding acc, I actually have two models so I modified the snippet but made a mistake. They are all <code>acc</code>.</p>

4gatepylon: <p>Resolved! If you look at the code inside evaluate you see that the total_number and total_correct are initialized in the wrong location. This means that every run it returns the value for a different batch (which is probably why it’s cyclic: it starts at different batches in some sort of cycle). The accuracies we were seeing were inside the batch.</p>