Can we perform the training and evaluation of a model separately?

joey.s: <p>Hi, I have some questions:</p>
<p>1- Can I perform model.train() and model.eval() separately, each of them in an independent script? or this will affect the model performance and they have to be called respectively?</p>
<p>2- In case that we can perform only the training: after saving a checkpoint for each epoch during the training, can we load these checkpoints from the checkpoints file to perform the evaluation and plot the evaluation performance?</p>

ptrblck: <ol>
<li>
<p>Yes, you can write own scripts for the training and evaluation of the model. Make sure to load the <code>state_dict</code> properly before starting the evaluation.</p>
</li>
<li>
<p>Yes, that should be possible.</p>
</li>
</ol>

joey.s: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, thanks for confirming that for me. So, after the training and saving the checkpoint as follow:</p>
<pre><code class="lang-auto">## Save checkpoint ##
        PATH = 'checkpoints/'+str(index)+'.pth'
        torch.save({
            'epoch': index + 1,
            'valid_loss': valid_loss,
            'model_state_dict': model.state_dict(),
            'optimizer_state_dict': optimizer.state_dict(),
            }, PATH)
</code></pre>
<p>I should add optimizer.zero_grad() and optimizer.step() before loading the checkpoints, is that right?</p>

ptrblck: <p>I would assume your validation script wouldn’t have an optimizer so I’m unsure where you would call <code>zero_grad()</code> and <code>step()</code>. Also, if you are calling these operations right after another without a <code>backward</code> operation, the <code>step()</code> would only change the parameters if running stats were used so I’m also unsure why these operations should be executed.</p>

joey.s: <p>When I load the checkpoint to evaluate the model and without using these operations, I got this error:</p>
<pre><code class="lang-auto">UserWarning: Detected call of `lr_scheduler.step()` before `optimizer.step()`. In PyTorch 1.1.0 and later, you should call them in the opposite order: `optimizer.step()` before `lr_scheduler.step()’
</code></pre>
<p>before printing the performance. So, I tried to execute them as in training prosses.</p>
<pre><code class="lang-auto">test_pred = model(test_data)
optimizer.zero_grad()

test_loss = model_fit(test_pred, test_label)
optimizer.step()
</code></pre>
<p>Doing this removes the warning error but the results seem to be similar. So, do I ignore the warning message or keep using these operations?</p>

ptrblck: <p>Using <code>model_fit</code> on the validation dataset seems wrong and I assume that you might be optimizing the model internally.<br>
If so, make sure to calculate the forward passes only to get the predictions and to compute the accuracy of the model. No learning rate scheduler or optimizer would be needed to do so.</p>

joey.s: <p>Sorry, <code>model_fit</code> is only a custom function to do the following: <code>loss = F.nll_loss(x_pred, x_output, ignore_index)</code>. So, what I understand is that I do not need to call <code>scheduler.step()</code> when performing the evaluation process.</p>

ptrblck: <p>Yes, you don’t even need to create a learning rate scheduler and an optimizer in your evaluation script since you are not training the model at all.</p>

joey.s: <p>Thanks. It really helps.</p>