Training is slowed down dramatically when evaluating validation set

zacheberhart: <p>I’m fairly new to PyTorch and am playing around with training on a GPU for the first time.</p>
<p>I’m using a pre-trained Densenet121. Unfrozen layers:</p>
<pre><code class="lang-auto">Sequential(
  (fc1): Linear(in_features=1024, out_features=500, bias=True)
  (relu): ReLU()
  (fc2): Linear(in_features=500, out_features=81, bias=True)
  (output): Softmax()
)
</code></pre>
<p>Anyway, training is working fine (though still fairly slow considering) but <strong>when I starting calculating the Validation Loss and Accuracy, the training slows down dramatically</strong>. Maybe 3-4X slower. Is there anything I am doing wrong to cause this? What might I do to improve training speed with validation?</p>
<p><strong>Code below:</strong></p>
<pre><code class="lang-auto">criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.classifier.parameters(), lr = 0.001)

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
torch.cuda.is_available()

# in-training validation
def validation(model, testloader, criterion):
    test_loss = 0
    accuracy = 0
    for inputs, labels in testloader:
        inputs, labels = inputs.to(device), labels.to(device)

        outputs = model.forward(inputs)
        test_loss += criterion(outputs, labels).item()

        ps = torch.exp(outputs)
        equality = (labels.data == ps.max(dim=1)[1])
        accuracy += equality.type(torch.cuda.FloatTensor).mean()
    
    return test_loss, accuracy

model.to(device)

for epoch in range(epochs):
    running_loss = 0
    model.train()

    for ii, (inputs, labels) in enumerate(trainloader):
        steps += 1
        optimizer.zero_grad()
        inputs, labels = inputs.to(device), labels.to(device)
        outputs = model.forward(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()
        running_loss += loss.item()
        
        if steps % print_steps == 0:
            model.eval()
            with torch.no_grad():
                test_loss, accuracy = validation(model, testloader, criterion)
            
            print("Epoch: {}/{}.. ".format(epoch+1, epochs),
                  "Training Loss: {:.3f}.. ".format(running_loss / print_steps),
                  "Test Loss: {:.3f}.. ".format(test_loss / len(testloader)),
                  "Test Accuracy: {:.3f}".format(accuracy / len(testloader)))
            
            running_loss = 0
            model.train()
</code></pre>

ptrblck: <p>Some minor issues:<br>
You should call the model directly like this <code>model(inputs)</code> instead of <code>model.forward(inputs)</code>. However, that shouldn’t be the issue regarding the performance.<br>
It seems your model returns log probabilities. You can directly get the prediction using <code>torch.argmax(outputs, dim=1)</code> instead of using <code>torch.exp</code>.</p>

Shubham_Singh1: <p>i am training deberta-v3-large, for training,  average speed is (5it/s) but for validation average speed is (1 it/s) which is 5 times slower than training, i am wondering why it’s happening.<br>
I am using dynamic padding(means for validation set examples has more tokens), could this be the reason, but i am creating 5 folds, so i think dynamic padding couldn’t be the reason.<br>
thanks!</p>

ptrblck: <aside class="quote no-group" data-username="Shubham_Singh1" data-post="3" data-topic="24917">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shubham_singh1/48/48889_2.png" class="avatar"> Shubham_Singh1:</div>
<blockquote>
<p>I am using dynamic padding(means for validation set examples has more tokens), could this be the reason, but i am creating 5 folds, so i think dynamic padding couldn’t be the reason.</p>
</blockquote>
</aside>
<p>I’m not sure what “5 folds” would mean in this case and how the number of tokens relate to the model (are there more indices used in the inputs?) so could you add more details to your use case?<br>
It might also help to profile the workflow with the PyTorch profiler or e.g. Nsight System.</p>

Shubham_Singh1: <p>Thanks for your response, by 5 fold means i am splitting my data in 5 equal part and then training model on 4parts and evaluating on remaining 1 part, in this way i am training 5 models so that each of the 5 parts acts as validation set once, by token length  i misunderstood model size by token length i.e (deberta-v3-large will take more time for training and evaluation  as compared to deberta-large or deberta-v3-base) as model size is less so we can choose slightly bigger batch_size which will speedup training/validation little bit, so i misinterpreted it as lesser the token length faster the speed.<br>
If you want then i can share my code along with the data.<br>
thanks!</p>