Why the weight still change?

username: ForeverZH0204
cooked: <p>I came into sth that really strange, this is my code</p>
<pre><code>def model_eval(model):
    model_the_wrong_images = []
    model_the_real_label = []
    model_the_pred = []
    correct = 0
    total = 0
    with torch.no_grad():  
        model.eval()
        for idx, test in enumerate(test_loader):

            image, label = test
            total += image.size()[0]

            pred = torch.argmax(model(image), 1)
            correct += (pred ==  label).sum().item()
        
            if idx == 4:
                #print(model(image))
                print(torch.argmax(model(image), 1))
                print(pred)
                print(torch.argmax(model(image), 1))
                print(pred == torch.argmax(model(image), 1))
                print(correct)
                print(label)

            for i in range(len(image)):
                if not (pred == label)[i].item():
                    model_the_wrong_images.append(image[i])
                    model_the_real_label.append(label[i])
                    model_the_pred.append(torch.argmax(model(image[i].unsqueeze_(0)), 1))

    print(correct)
    print(total)
    print('The accuracy on testset is : %f'%(correct / total))
    
    return model_the_wrong_images, model_the_real_label, model_the_pred
</code></pre>
<p>I write this func to eval my model, but i found that once i repeat the eval process with the model that has already been trained, the accuracy changes everytime.</p>
<p><strong>Why my model’s params can’t keep const??? I wrote that <code>model.eval()</code>, but it seems no work</strong></p>
<p>So i print the output and found the output change.Go futrher, i print the output of the 4th subset of the test process to see, something terrible happen, the outputs of the <code>if</code> are:</p>
<pre><code>tensor([9, 7, 2, 4])
tensor([9, 7, 3, 4])
tensor([9, 7, 3, 4])
tensor([1, 1, 1, 1], dtype=torch.uint8)
17
tensor([9, 7, 3, 4])
</code></pre>
<p>How could the <code>pred</code> and <code>torch.argmax(model(image), 1)</code> become two different things?I get pred by <code>pred = torch.argmax(model(image), 1)</code>!</p>
<p>Who can tell me why!  Thanks a lot!!!</p>

username: ForeverZH0204
cooked: <p>Anyone Can help me?<img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=6" title=":joy:" class="emoji" alt=":joy:"><img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=6" title=":joy:" class="emoji" alt=":joy:"><img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=6" title=":joy:" class="emoji" alt=":joy:"><img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=6" title=":joy:" class="emoji" alt=":joy:"></p>

username: ashwin.raju93
cooked: <p>did you try setting <code>torch.backends.cudnn.deterministic = True</code></p>

username: ForeverZH0204
cooked: <p>Thanks!<br>
I run the script on cpu. Do i need to do any change to the code?</p>

username: ForeverZH0204
cooked: <p>Still no use…<br>
So frustrated</p>

username: Naruto-Sasuke
cooked: <p>I think the direct way to see whether the model changes is to print the parameters out.</p>

username: ForeverZH0204
cooked: <p>As you suggest, i found the params seems keep const, but the eval process always get different result</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/6/6aede08ac4acc63f3e16184270330ac7c173c8f3.png" data-download-href="https://discuss.pytorch.org/uploads/default/6aede08ac4acc63f3e16184270330ac7c173c8f3" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/6/6aede08ac4acc63f3e16184270330ac7c173c8f3_2_690x113.png" alt="image" data-base62-sha1="ffWgcd69OsFGrHS4CpKtTGEXEzN" width="690" height="113" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/6/6aede08ac4acc63f3e16184270330ac7c173c8f3_2_690x113.png, https://discuss.pytorch.org/uploads/default/optimized/2X/6/6aede08ac4acc63f3e16184270330ac7c173c8f3_2_1035x169.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/6/6aede08ac4acc63f3e16184270330ac7c173c8f3.png 2x" data-dominant-color="F5F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1224×202 14.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: bharat0to
cooked: <p>Can you post the full code, so that I can see if you are using any layers that has some randomization, and you might also be using test time augmentation(TTA)</p>

username: Naruto-Sasuke
cooked: <p>Set  <code>shuffle=False</code> and no_flip or something if you add flipping as a kind of data augmentation.</p>

username: ForeverZH0204
cooked: <pre><code>import torch
import torchvision
transform = torchvision.transforms.Compose(
                    [torchvision.transforms.ToTensor(),
                     torchvision.transforms.Normalize([0.5, 0.5, 0.5],[0.5, 0.5, 0.5])])
trainset = torchvision.datasets.MNIST(download=True, root='./data/MNIST',
                                    train=True,transform=transform)
testset = torchvision.datasets.MNIST(download=True, root='./data/MNIST',
                                    train=False,transform=transform)
train_loader = torch.utils.data.DataLoader(dataset=trainset, shuffle=False,
                            batch_size=64)
test_loader = torch.utils.data.DataLoader(dataset=testset, shuffle=False,
                            batch_size=4)
import torch.nn.functional as F
import torch.nn as nn

class Conv_Classifier(nn.Module):
    def __init__(self):
        super(Conv_Classifier, self).__init__()
        self.conv1 = nn.Conv2d(1, 5, 5)
        self.pool1 = nn.MaxPool2d(2)
        self.conv2 = nn.Conv2d(5, 16, 5)
        self.pool2 = nn.MaxPool2d(2)
        self.fc1 = nn.Linear(256, 20)
        self.fc2 = nn.Linear(20, 10)

    def forward(self, x):
        x = F.relu(self.pool1((self.conv1(x))))
        x = F.relu(self.pool2((self.conv2(x))))
        x = F.dropout2d(x)
        x = x.view(-1, 256)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        return x

Mnist_Classifier = Conv_Classifier()

def model_train(model, optimizer):
    total_loss = 0
    model.train()
    for i in range(5):

        for idx, data in enumerate(train_loader):
            image, label = data
        
            optimizer.zero_grad()
            pred = model(image)
            loss = loss_fn(pred, label)
            loss.backward()
            optimizer.step()
        
            total_loss += loss
            if idx % 149 == 0:
                print('[epoch %(epoch)d batch %(batch)d] loss : '%{'epoch':i, 'batch':idx}, loss)
        print('Total loss after %(epoch)d epoch'%{'epoch':i}, total_loss)
        total_loss=0

model_train(Mnist_Classifier, torch.optim.SGD(Mnist_Classifier.parameters(),lr=0.01, momentum=0.9))

the_wrong_images = []
the_real_label = []
the_pred = []

def model_eval(model):
    model_the_wrong_images = []
    model_the_real_label = []
    model_the_pred = []
    correct = 0
    total = 0
    with torch.no_grad():  
        model.eval()
        for idx, test in enumerate(test_loader):

            image, label = test
            total += image.size()[0]

            pred = torch.argmax(model(image), 1)
            correct += (pred ==  label).sum().item()
      
    
        
            for i in range(len(image)):
                if not (pred == label)[i].item():
                    model_the_wrong_images.append(image[i])
                    model_the_real_label.append(label[i])
                    model_the_pred.append(torch.argmax(model(image[i].unsqueeze_(0)), 1))

    print(correct)
    print(total)
    print('The accuracy on testset is : %f'%(correct / total))
    
    return model_the_wrong_images, model_the_real_label, model_the_pred

the_wrong_images, the_real_label, the_pred = model_eval(Mnist_Classifier)
</code></pre>
<p>It is just a simple example on Mnist, but if you repeat the model_eval func, you will find the accuracy will change…</p>

username: ptrblck
cooked: <p>Since you are using the functional API for <code>F.dropout</code>, it will be always set to <code>training</code>, i.e. it’ll always drop your activations.<br>
You could use the module API <code>nn.Dropout(p=0.5)</code> or alternatively pass the <code>self.training</code> flag to your current code:</p>
<pre><code class="lang-python">x = F.dropout(x, training=self.training)
</code></pre>

username: ForeverZH0204
cooked: <p>I did, but the output will change even for the same input</p>
<p>My code is below, so sad</p>

username: ForeverZH0204
cooked: <p>Thank you!<br>
My problem is solved<br>
Really a foolish mistake<br>
Wish u a good day!</p>

username: achm3dz
cooked: <p>what was the mistake exactly, if you remember?</p>