Model.eval() accuracy is 0 and running_corrects is 0

lima: <p>I’m having an issue with my DNN model.<br>
During train phase, the accuracy is 0.968 and the loss is 0.103, but during test phase with model.eval(), the accuracy is 0 and the running corrects is 0.</p>
<pre><code class="lang-auto">def train(model, device, train_loader, criterion, optimizer, scheduler, epoch, iter_meter, experiment):
    model.train()
    liveloss = PlotLosses()
    data_len = len(train_loader.dataset)
    with experiment.train():
        
        logs = {}
        running_loss = 0.0
        running_corrects = 0
        
        for batch_idx, _data in enumerate(train_loader):
            features, labels = _data[:][:,:,:-1], _data[..., -1]
            features = features.permute(0, 2, 1)

            features, labels = features.to(device), labels.to(device) 
            
            optimizer.zero_grad()

            output = model(features) 
            loss = criterion(output, torch.max(labels, 1)[1])
            loss.backward()

            experiment.log_metric('loss', loss.item(), step=iter_meter.get())
            experiment.log_metric('learning_rate', scheduler.get_last_lr(), step=iter_meter.get())

            optimizer.step()
            scheduler.step()
            iter_meter.step()
            
            _, preds = torch.max(output, 1)
            running_loss += loss.detach() * features.size(0)
            running_corrects += torch.sum(preds == torch.max(labels, 1)[1])
        epoch_loss = running_loss / len(train_loader.dataset)
        epoch_acc = running_corrects.float() / len(train_loader.dataset)
        logs['log loss'] = epoch_loss.item()
        logs['accuracy'] = epoch_acc.item()
    liveloss.update(logs)
    liveloss.send()


iter_meter = IterMeter()
for epoch in range(1, epochs + 1):
    train(model, device, train_loader, criterion, optimizer, scheduler, epoch, iter_meter, experiment)
    
</code></pre>
<p>The evaluation script is:</p>
<pre><code class="lang-auto">
def test(model, device, tst_loader, criterion, epoch, iter_meter, experiment):
    print('\nevaluating...')
    model.eval()
    test_loss = 0
    liveloss = PlotLosses()
    data_len = len(tst_loader.dataset)
    with experiment.test():
        with torch.no_grad():
        
            logs = {}
            running_loss = 0.0
            running_corrects = 0

            for batch_idx, _data in enumerate(tst_loader):
                features, labels = _data[:][:,:,:-1], _data[..., -1]
                features = features.permute(0, 2, 1)

                features, labels = features.to(device), labels.to(device)


                output = model(features) 
                loss = criterion(output, torch.max(labels, 1)[1])
                test_loss += loss.item() / len(tst_loader)

                experiment.log_metric('loss', loss.item(), step=iter_meter.get())

                iter_meter.step()

                _, preds = torch.max(output, 1)
                running_loss += loss.detach() * features.size(0)
                running_corrects += torch.sum(preds == torch.max(labels, 1)[1])
                
            epoch_loss = running_loss / len(tst_loader.dataset)
            epoch_acc = running_corrects.float() / len(tst_loader.dataset)
            logs['log loss'] = epoch_loss.item()
            logs['accuracy'] = epoch_acc.item()
    liveloss.update(logs)
    liveloss.send()

iter_meter = IterMeter()
for epoch in range(1, epochs + 1):
    test(model, device, tst_loader, criterion, epoch, iter_meter, experiment)
</code></pre>
<p>Is there something with the code?</p>

Henry_Chibueze: <p>I want to know what version of python this is.<br>
Some previous versions of python do not support +=<br>
This is not likely the issue tho coz it’ll flag an error.</p>
<p>Now I noticed u are not saving the model after training.<br>
U are passing the model into the training function as an argument and u are not returning it after training.<br>
When u then pass the model into the testing function, it doesn’t pass the already trained model.</p>
<p>So try making model a global variable so that whatever changes u make to it from the training function, it’ll reflect on the testing function.</p>
<p>Or</p>
<p>After training the model, return the already trained model and make model = train()</p>

lima: <p>Thank s for the reply. The += operator worked on model.train(). So I don’t think it’s the culprit.</p>

Henry_Chibueze: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/5/a5a323df27e57241b08df8d6c57778062f89ff7f.png" data-download-href="https://discuss.pytorch.org/uploads/default/a5a323df27e57241b08df8d6c57778062f89ff7f" title="main.py - traditional ML methods - Visual Studio Code 6_4_2021 11_32_06 AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/5/a5a323df27e57241b08df8d6c57778062f89ff7f_2_690x386.png" alt="main.py - traditional ML methods - Visual Studio Code 6_4_2021 11_32_06 AM" data-base62-sha1="nDij8Ko57d1E9Lo66KtampeTSph" width="690" height="386" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/5/a5a323df27e57241b08df8d6c57778062f89ff7f_2_690x386.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/5/a5a323df27e57241b08df8d6c57778062f89ff7f_2_1035x579.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/5/a5a323df27e57241b08df8d6c57778062f89ff7f.png 2x" data-dominant-color="232628"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">main.py - traditional ML methods - Visual Studio Code 6_4_2021 11_32_06 AM</span><span class="informations">1366×766 48.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Look at this image and correlate it with what my edit was talking about.<br>
when i printed x, it did not change because I only multiplied the x that I passed as a parameter of the ‘me’ function by 4 and not the original x.<br>
so x still remains 4.</p>

lima: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> can you help please?</p>

ptrblck: <p>I’m not sure about the background of the previous discussion about the inplace operation.</p>
<p>For your initial problem: are you seeing the same accuracy drop after you call <code>model.eval()</code> and use your test dataset for the sake of debugging?</p>

lima: <p>I didn’t get your point, but I call <code>model.train()</code> with the exact script above, on train data it shows accuracy is 0.968 and the loss is 0.103.<br>
Then I call <code>model.eval()</code>, and the accuracy is always 0.0000.<br>
Do I have to save the model first before I call <code>model.eval()</code>?</p>

ptrblck: <p>No, you don’t need to save the model before calling <code>eval()</code>.<br>
Are you seeing the performance drop on both, the training and validation dataset, after calling <code>model.eval()</code>?</p>

lima: <p>Sorry I’m a newbie. So I may be doing it all wrong.<br>
When I call <code>model.eval()</code> it just uses the test loader, so I’m not sure how the train performance could be affected.<br>
Please bear with me. Is there anything in the code above rhat looks off to you?<br>
Here’s what I’m doing.<br>
On jupyter norebook.<br>
I set the train loader and test loader.<br>
On one cell I run the train script<br>
Once train is done, I run the test script</p>

Henry_Chibueze: <p><a class="mention" href="/u/lima">@lima</a> <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
As I said earlier if you read my comment.<br>
You passed the model into the train function and trained it without returning the it after training.<br>
So the model you passed into the test function is not a trained model but rather a model that is yet to be trained.<br>
The training and weight updates only happened in the train function, so if u call the model any other place, it won’t have those trained weights and It’ll still be the untrained model it is…simply because you passed the model as a parameter to the train function and made the trained model peculiar to just the train function.</p>
<p>So you either make the model global and make the train and test function to access and use it</p>
<p>Or</p>
<p>You either return the trained model after you trained it in the train function and assign it’s call to a variable for inference</p>
<p>Or</p>
<p>You save the already trained model and load it when you want to make inference.</p>

Henry_Chibueze: <p><a class="mention" href="/u/lima">@lima</a> <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Make the model global so the train and test function can both access it without passing it individually as a parameter</p>
<p>Given that: model = the_model_that_you_declared_outside()</p>
<pre><code class="lang-auto">def train(device, train_loader, criterion, optimizer, scheduler, epoch, iter_meter, experiment):
    model.train()
    liveloss = PlotLosses()
    data_len = len(train_loader.dataset)
    with experiment.train():
        
        logs = {}
        running_loss = 0.0
        running_corrects = 0
        
        for batch_idx, _data in enumerate(train_loader):
            features, labels = _data[:][:,:,:-1], _data[..., -1]
            features = features.permute(0, 2, 1)

            features, labels = features.to(device), labels.to(device) 
            
            optimizer.zero_grad()

            output = model(features) 
            loss = criterion(output, torch.max(labels, 1)[1])
            loss.backward()

            experiment.log_metric('loss', loss.item(), step=iter_meter.get())
            experiment.log_metric('learning_rate', scheduler.get_last_lr(), step=iter_meter.get())

            optimizer.step()
            scheduler.step()
            iter_meter.step()
            
            _, preds = torch.max(output, 1)
            running_loss += loss.detach() * features.size(0)
            running_corrects += torch.sum(preds == torch.max(labels, 1)[1])
        epoch_loss = running_loss / len(train_loader.dataset)
        epoch_acc = running_corrects.float() / len(train_loader.dataset)
        logs['log loss'] = epoch_loss.item()
        logs['accuracy'] = epoch_acc.item()
    liveloss.update(logs)
    liveloss.send()


iter_meter = IterMeter()
for epoch in range(1, epochs + 1):
    train(device, train_loader, criterion, optimizer, scheduler, epoch, iter_meter, experiment)
    
#The evaluation script is:


def test(device, tst_loader, criterion, epoch, iter_meter, experiment):
    print('\nevaluating...')
    model.eval()
    test_loss = 0
    liveloss = PlotLosses()
    data_len = len(tst_loader.dataset)
    with experiment.test():
        with torch.no_grad():
        
            logs = {}
            running_loss = 0.0
            running_corrects = 0

            for batch_idx, _data in enumerate(tst_loader):
                features, labels = _data[:][:,:,:-1], _data[..., -1]
                features = features.permute(0, 2, 1)

                features, labels = features.to(device), labels.to(device)


                output = model(features) 
                loss = criterion(output, torch.max(labels, 1)[1])
                test_loss += loss.item() / len(tst_loader)

                experiment.log_metric('loss', loss.item(), step=iter_meter.get())

                iter_meter.step()

                _, preds = torch.max(output, 1)
                running_loss += loss.detach() * features.size(0)
                running_corrects += torch.sum(preds == torch.max(labels, 1)[1])
                
            epoch_loss = running_loss / len(tst_loader.dataset)
            epoch_acc = running_corrects.float() / len(tst_loader.dataset)
            logs['log loss'] = epoch_loss.item()
            logs['accuracy'] = epoch_acc.item()
    liveloss.update(logs)
    liveloss.send()

iter_meter = IterMeter()
for epoch in range(1, epochs + 1):
    test(device, tst_loader, criterion, epoch, iter_meter, experiment)
    
</code></pre>
<p>Or you can return the model after training in the train function like so:</p>
<pre><code class="lang-auto">def train(model, device, train_loader, criterion, optimizer, scheduler, epoch, iter_meter, experiment):
    model.train()
    liveloss = PlotLosses()
    data_len = len(train_loader.dataset)
    with experiment.train():
        
        logs = {}
        running_loss = 0.0
        running_corrects = 0
        
        for batch_idx, _data in enumerate(train_loader):
            features, labels = _data[:][:,:,:-1], _data[..., -1]
            features = features.permute(0, 2, 1)

            features, labels = features.to(device), labels.to(device) 
            
            optimizer.zero_grad()

            output = model(features) 
            loss = criterion(output, torch.max(labels, 1)[1])
            loss.backward()

            experiment.log_metric('loss', loss.item(), step=iter_meter.get())
            experiment.log_metric('learning_rate', scheduler.get_last_lr(), step=iter_meter.get())

            optimizer.step()
            scheduler.step()
            iter_meter.step()
            
            _, preds = torch.max(output, 1)
            running_loss += loss.detach() * features.size(0)
            running_corrects += torch.sum(preds == torch.max(labels, 1)[1])
        epoch_loss = running_loss / len(train_loader.dataset)
        epoch_acc = running_corrects.float() / len(train_loader.dataset)
        logs['log loss'] = epoch_loss.item()
        logs['accuracy'] = epoch_acc.item()
    liveloss.update(logs)
    liveloss.send()
    return model


iter_meter = IterMeter()
for epoch in range(1, epochs + 1):
   model = train(model, device, train_loader, criterion, optimizer, scheduler, epoch, iter_meter, experiment)
    
#The evaluation script is:


def test(model, device, tst_loader, criterion, epoch, iter_meter, experiment):
    print('\nevaluating...')
    model.eval()
    test_loss = 0
    liveloss = PlotLosses()
    data_len = len(tst_loader.dataset)
    with experiment.test():
        with torch.no_grad():
        
            logs = {}
            running_loss = 0.0
            running_corrects = 0

            for batch_idx, _data in enumerate(tst_loader):
                features, labels = _data[:][:,:,:-1], _data[..., -1]
                features = features.permute(0, 2, 1)

                features, labels = features.to(device), labels.to(device)


                output = model(features) 
                loss = criterion(output, torch.max(labels, 1)[1])
                test_loss += loss.item() / len(tst_loader)

                experiment.log_metric('loss', loss.item(), step=iter_meter.get())

                iter_meter.step()

                _, preds = torch.max(output, 1)
                running_loss += loss.detach() * features.size(0)
                running_corrects += torch.sum(preds == torch.max(labels, 1)[1])
                
            epoch_loss = running_loss / len(tst_loader.dataset)
            epoch_acc = running_corrects.float() / len(tst_loader.dataset)
            logs['log loss'] = epoch_loss.item()
            logs['accuracy'] = epoch_acc.item()
    liveloss.update(logs)
    liveloss.send()

iter_meter = IterMeter()
for epoch in range(1, epochs + 1):
    test(model, device, tst_loader, criterion, epoch, iter_meter, experiment)
</code></pre>

lima: <p><a class="mention" href="/u/henry_chibueze">@Henry_Chibueze</a> <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>I think I can pinpoint the issue but I still can’t get it solved.<br>
The problem is I don’t know how to get the predictions to look like the labels.</p>
<pre><code class="lang-auto">test_acc = 0.0
for _data in loaders['test']:
    with torch.no_grad():
        data, target = _data[:][:,:,:-1], _data[..., -1]
        data = data.permute(0, 2, 1)
        data, target = data.to(device), target.to(device)
        output = trained_model(data)
        # calculate accuracy
        _, pred = torch.max(output, dim=1) 

        correct = torch.sum(pred.eq(torch.max(target, 1)[1]))
        print(f'prediction: {pred}')
        print(f'label: {(torch.max(target, 1)[1])}')

        test_acc += torch.mean(correct.float())
print('Accuracy of the network on {} test frames: {}%'.format(len(tst_data3), round(test_acc.item()*100.0/len(loaders['test']), 2)))
</code></pre>
<p>If I print the prediction and the label, I get different values:</p>
<pre><code class="lang-auto">print(f'prediction: {pred}')

prediction: tensor([194,  86, 492, 492, 132, 132, 263, 216, 241,  17, 263, 216, 127, 399,
        492, 500, 420, 263, 390, 510, 216, 510, 132, 194, 263, 217, 263,  23,
        216, 395, 132, 297, 390, 194, 263, 492, 114, 216, 194, 503,  20, 217,
        297, 477, 476, 263, 263, 479, 466, 500, 132, 263, 361, 194,  92, 510,
        393, 216, 500,  53, 194, 510, 216, 216, 269, 216, 228, 194, 119, 415,
        477, 477, 114, 477, 476, 477, 503, 194, 170, 216, 263, 194, 221, 503,
        263, 466, 263, 114, 263, 492,  46, 449, 286, 286, 263, 132, 420, 406,
        492, 194, 263, 194, 263, 263, 194, 477, 114, 119, 477, 510, 241, 492,
        286, 263, 170, 216, 216, 194, 127, 263, 492, 477, 216, 114, 194, 360,
        479, 390], device='cuda:0')
</code></pre>
<pre><code class="lang-auto">print(f'label: {(torch.max(target, 1)[1])}')

label: tensor([3, 0, 0, 0, 1, 0, 6, 0, 2, 0, 5, 0, 1, 0, 4, 0, 0, 0, 0, 0, 0, 4, 0, 0,
        3, 0, 4, 2, 0, 6, 0, 0, 3, 0, 0, 3, 0, 0, 0, 1, 0, 0, 5, 0, 0, 6, 0, 0,
        0, 6, 0, 5, 0, 0, 6, 0, 1, 0, 4, 0, 0, 4, 0, 1, 0, 1, 0, 0, 0, 4, 0, 0,
        0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 0, 4, 0, 0, 6, 0, 0, 5, 0,
        0, 0, 1, 0, 3, 0, 3, 4, 0, 5, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0, 0, 3, 0,
        0, 0, 1, 0, 1, 1, 0, 6], device='cuda:0')
</code></pre>

ptrblck: <aside class="quote no-group" data-username="Henry_Chibueze" data-post="10" data-topic="123271">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/henry_chibueze/48/28458_2.png" class="avatar"> Henry_Chibueze:</div>
<blockquote>
<p>The training and weight updates only happened in the train function, so if u call the model any other place, it won’t have those trained weights and It’ll still be the untrained model it is…simply because you passed the model as a parameter to the train function and made the trained model peculiar to just the train function.</p>
</blockquote>
</aside>
<p>I don’t think that’s correct as seen e.g. here:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
import torch.nn as nn

def train(model, optimizer):
    for epoch in range(10):
        optimizer.zero_grad()
        output = model(torch.randn(1, 1))
        loss = criterion(output, torch.randn(1, 1))
        loss.backward()
        optimizer.step()
        print('epoch {}, loss {}, weight {}'.format(
            epoch, loss.item(), model.weight))

if __name__=='__main__':
    model = nn.Linear(1, 1)
    print('before training')
    print(model.weight)
    
    optimizer = torch.optim.SGD(model.parameters(), lr=1.)
    criterion = nn.MSELoss()
    
    train(model, optimizer)
    print('after training')
    print(model.weight)
</code></pre>
<p>As you can see, the model is created in the if-clause guard, passed to the train function, and is not returned.<br>
Nevertheless, the weight updates will be performed inplace on the model reference and thus the model in the same scope will contain the upgraded parameters.</p>