Why running an extra forward function with random generated input but no loss calculation or backpropagation in the previous normal training process can change the training result?

username: John_M
cooked: <p>I am trying a new training method (semi supervise GAN) but there is some result I don’t understand. The problem seems to be some Pytorch feature I don’t understand, but not in semi supervise GAN.<br>
The classifier is the EfficientNet_b0.</p>
<p>for epoch in range(1, epochs + 1):<br>
print (’\nEpoch: {} '.format(epoch))</p>
<pre><code>for idx, dataLabeled in enumerate(train_loader):
    classifier.train()
    optimizerC.zero_grad()

    # I am working on these 3 lines that cause strange results
    # These 3 lines is a small part of some semi supervise GAN but not completed.    
    noise = torch.randn(BATCH_SIZE, 100, 1, 1, device=device)
    generated = (netG(noise)+1.0)/2.0
    # outputG = classifier(generated.detach())    

    labeled_data, labels = dataLabeled
    labeled_data, labels = labeled_data.to(device), labels.to(device)

    output = classifier(labeled_data)
    labeled_loss = criterion(output, labels) 
    labeled_loss.backward()    
    optimizerC.step()

correct = 0
num_test = len(test_loader.dataset)
# testing accuracy, I don't think the problem is here 
with torch.no_grad():
    for batch in test_loader:
        image, labelS = batch
        data = image.to(device)
        target = labelS.to(device)
        output = model(data)
        test_loss += criterion(output, target).item() 
        pred = output.argmax(dim=1, keepdim=True)
        correct += pred.eq(target.view_as(pred)).sum().item()
    print('\nTest set: Accuracy: {}/{} ({:.4f}%)\n'.format(correct, num_test, 100. * correct / num_test))
</code></pre>
<p>After I commented out the line # outputG = classifier(generated.detach()), it worked just like the EfficientNet. I tested it several times and the results were similar.<br>
Epoch: 1    Test set: Accuracy: 114/163 (69.9387%)<br>
Epoch: 2    Test set: Accuracy: 123/163 (75.4601%)<br>
Epoch: 3    Test set: Accuracy: 131/163 (80.3681%)<br>
Epoch: 4    Test set: Accuracy: 126/163 (77.3006%)</p>
<p>If I uncommented this line, the results changed. The accuracy keeps going down every time I try. The only difference is adding this line # outputG = classifier(generated.detach()).<br>
Epoch: 1    Test set: Accuracy: 99/163 (60.7362%)<br>
Epoch: 2    Test set: Accuracy: 117/163 (71.7791%)<br>
Epoch: 3    Test set: Accuracy: 106/163 (65.0307%)<br>
Epoch: 4    Test set: Accuracy: 97/163 (59.5092%)</p>
<p>So why this one line outputG = classifier(generated.detach()) can make the result changed? It’s just a forward function. I don’t even run loss calculation or backpropagation for it.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="John_M" data-post="1" data-topic="150163">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/john_m/48/46113_2.png" class="avatar"> John_M:</div>
<blockquote>
<p>So why this one line outputG = classifier(generated.detach()) can make the result changed? It’s just a forward function. I don’t even run loss calculation or backpropagation for it.</p>
</blockquote>
</aside>
<p>Some layer, such as batchnorm layers, will perform updates of stats during the forward pass.<br>
In the example of batchnorm layers (which are used in <code>EfficientNet_b0</code>), the running stats will be updated using the current batch statistics and will thus change the validation results.</p>