Different training accuracies using same random seed

sr_frost: <p>Hi,<br>
I am trying to evaluate my model on the whole training set after each epoch.<br>
This is what I did:</p>
<pre><code class="lang-auto">torch.manual_seed(1)
model = ConvNet(num_classes=num_classes)
cost_fn = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)  

def compute_accuracy(model, data_loader):
    correct_pred, num_examples = 0, 0
    for features, targets in data_loader:
        logits = model(features)
        predicted_labels = torch.argmax(logits, 1)
        num_examples += targets.size(0)
        correct_pred += (predicted_labels == targets).sum()
    return correct_pred.float()/num_examples * 100
    
for epoch in range(num_epochs):
    model = model.train()
    for features, targets in train_loader:
        logits = model(features)
        cost = cost_fn(logits, targets)
        optimizer.zero_grad()
        cost.backward()
        optimizer.step()
        
    model = model.eval()
    print('Epoch: %03d/%03d training accuracy: %.2f%%' % (
          epoch+1, num_epochs, 
          compute_accuracy(model, train_loader)))
</code></pre>
<p>the output was convincing:</p>
<pre><code class="lang-auto">Epoch: 001/005 training accuracy: 89.08%
Epoch: 002/005 training accuracy: 90.41%
Epoch: 003/005 training accuracy: 91.70%
Epoch: 004/005 training accuracy: 92.31%
Epoch: 005/005 training accuracy: 92.95%
</code></pre>
<p>But then I added another line at the end of the training loop, to also evaluate the model on the whole test set after each epoch:</p>
<pre><code class="lang-auto">for epoch in range(num_epochs):
    model = model.train()
    for features, targets in train_loader:
        logits = model(features)
        cost = cost_fn(logits, targets)
        optimizer.zero_grad()
        cost.backward()
        optimizer.step()
        
    model = model.eval()
    print('Epoch: %03d/%03d training accuracy: %.2f%%' % (
          epoch+1, num_epochs, 
          compute_accuracy(model, train_loader)))
    print('\t\t testing accuracy: %.2f%%' % (compute_accuracy(model, test_loader)))
</code></pre>
<p>But the training accuracies started to change:</p>
<pre><code class="lang-auto">Epoch: 001/005 training accuracy: 89.08%
		 testing accuracy: 87.66%
Epoch: 002/005 training accuracy: 90.42%
		 testing accuracy: 89.04%
Epoch: 003/005 training accuracy: 91.84%
		 testing accuracy: 90.01%
Epoch: 004/005 training accuracy: 91.86%
		 testing accuracy: 89.83%
Epoch: 005/005 training accuracy: 92.45%
		 testing accuracy: 90.32%
</code></pre>
<p>Am I doing something wrong? I expected the training accuracies to remain the same because the manual seed is 1 in both cases.<br>
Is this an expected output ?</p>

InnovArul: <p>The test code that you have added at the end may use random generator which influences the training data selection from epoch-2. If you really want the previous behavior, Can you save and set the random number generator state before and after your test code?<br>
using <a href="https://pytorch.org/docs/stable/torch.html#torch.get_rng_state" rel="nofollow noopener"> <code>torch.get_rng_state</code> </a>and  <a href="https://pytorch.org/docs/stable/torch.html#torch.set_rng_state" rel="nofollow noopener"> <code>torch.set_rng_state</code> </a>.</p>
<pre><code class="lang-auto">rng_state = torch.get_rng_state()
print('\t\t testing accuracy: %.2f%%' % (compute_accuracy(model, test_loader)))
torch.set_rng_state(rng_state)
</code></pre>

sr_frost: <p>Thanks. It worked. <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>
<p>But I think last line was meant to be :</p>
<pre><code class="lang-auto">torch.set_rng_state(rng_state)
</code></pre>
<aside class="quote no-group quote-modified" data-username="InnovArul" data-post="2" data-topic="58663">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/innovarul/48/5282_2.png" class="avatar"> InnovArul:</div>
<blockquote>
<p>torch.get_rng_state(rng_state)</p>
</blockquote>
</aside>

InnovArul: <p>you are right. sorry for the typo.</p>