Testing in loop as training

Faraz: <p>I am new to PyTorch can someone please have a look if I am doing it right? I am trying to run validation in same epoch as training.</p>
<pre><code class="lang-auto">for epoch in range(num_epochs):
    logs = {}
    total_correct = 0
    total_loss = 0
    total_images = 0
    total_val_loss = 0

    model.train()
    for i, (data, target) in enumerate(train_loader):

        images = Variable(data).to('cuda:0')
        labels = Variable(target).to('cuda:0')
        optimizer.zero_grad()

        # Forward propagation
        outputs = model(images)

        # Calculating loss with softmax to obtain cross entropy loss

        # loss = criterion(outputs, labels)

        loss = criterion(outputs, labels)    #....&gt;


        # Backward prop
        loss.backward()

        # Updating gradients
        optimizer.step()

        # Total number of labels
        total_images+= labels.size(0)


        # Obtaining predictions from max value
        _, predicted = torch.max(outputs.data, 1)



        # Calculate the number of correct answers
        correct = (predicted == labels).sum().item()

        total_correct+=correct
        total_loss+=loss.item()
    logs['log loss'] = total_loss / total_images
    logs['Accuracy'] = ((total_correct / total_images) * 100)


    print('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}, Accuracy: {:.2f}%'
              .format(epoch + 1, num_epochs, i + 1, len(test_loader), (total_loss / total_images),
                      (total_correct / total_images) * 100))




    # Testing the model

    with torch.no_grad():
        correct = 0
        total = 0
        total_losss =0
        model.eval()
        for data, target in test_loader:
            images = Variable(data).to('cuda:0')
            labels = Variable(target).to('cuda:0')
            outputs = model(images)

            _, predicted = torch.max(outputs.data, 1)
            total += labels.size(0)
            correct += (predicted == labels).sum().item()
            total_losss += loss.item()

            accuracy = correct / total

        print('Test Accuracy of the model: {} %'.format(100 * correct / total))
        logs['val_' + 'log loss'] = total_losss/total
        logs['val_' + 'Accuracy'] = ((correct / total) * 100)
       # logs['accuracy'] = accuracy
    liveloss.update(logs)
    liveloss.draw()
</code></pre>

albanD: <p>Hi,</p>
<p>A few things:</p>
<ul>
<li>
<code>Variable</code> is not needed anymore, you can have simply <code>images = data.to('cuda:0')</code>
</li>
<li>You are missing the <code>optimizer.zero_grad()</code> before the backward ! You need to manually reset the weights to 0 when you pytorch (see discussion about this here: <a href="https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903" class="inline-onebox">Why do we need to set the gradients manually to zero in pytorch?</a> )</li>
<li>You should not use <code>.data</code>. If you want to compute things without tracking history, you can either use detach() as <code>_, predicted = torch.max(outputs.detach(), 1)</code> or wrap the computations in <code>with torch.no_grad():</code> to compute predicted and correct.</li>
<li>You’re doing the right thing with <code>.item()</code> to accumulate the loss.</li>
<li>For the evaluattion, same thing about .data and Variable</li>
<li>You might be missing a <code>model.eval()</code> before the evaluation loop.</li>
</ul>

Faraz: <p>Thankyou so much, I have modified the code based on your suggestion as such:</p>
<pre><code class="lang-auto">for epoch in range(num_epochs):
    logs = {}
    total_correct = 0
    total_loss = 0
    total_images = 0
    total_val_loss = 0

    model.train()
    for i, (data, target) in enumerate(train_loader):

        images = data.to('cuda:0')
        labels = target.to('cuda:0')


        # Forward propagation
        outputs = model(images)

        # Calculating loss with softmax to obtain cross entropy loss

        # loss = criterion(outputs, labels)

        loss = criterion(outputs, labels)    #....&gt;

        optimizer.zero_grad()
        # Backward prop
        loss.backward()

        # Updating gradients
        optimizer.step()

        # Total number of labels
        total_images+= labels.size(0)


        # Obtaining predictions from max value
        _, predicted = torch.max(outputs.detach(), 1)



        # Calculate the number of correct answers
        correct = (predicted == labels).sum().item()

        total_correct+=correct
        total_loss+=loss.item()
    logs['log loss'] = total_loss / total_images
    logs['Accuracy'] = ((total_correct / total_images) * 100)


    print('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}, Accuracy: {:.2f}%'
              .format(epoch + 1, num_epochs, i + 1, len(test_loader), (total_loss / total_images),
                      (total_correct / total_images) * 100))




    # Testing the model
    model.eval()

    with torch.no_grad():
        correct = 0
        total = 0
        total_losss =0

        for data, target in test_loader:
            images = data.to('cuda:0')
            labels = target.to('cuda:0')
            outputs = model(images)

            _, predicted = torch.max(outputs.detach(), 1)
            total += labels.size(0)
            correct += (predicted == labels).sum().item()
            total_losss += loss.item()

            accuracy = correct / total

        print('Test Accuracy of the model: {} %'.format(100 * correct / total))
        logs['val_' + 'log loss'] = total_losss/total
        logs['val_' + 'Accuracy'] = ((correct / total) * 100)
       # logs['accuracy'] = accuracy
    liveloss.update(logs)
    liveloss.draw()
</code></pre>

albanD: <p>Yes, that looks good <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
Small note is that you don’t need the <code>.detach()</code> for the evaluation as you’re in no_grad mode anyway. But won’t change much !</p>