Continue trainning after saving model

Najeh_Nafti: <p>How can I save best model weights to continue training the model after stopping because of limited GPU resources?</p>

ptrblck: <p>The ImageNet example would be a good reference for resuming the training.<br>
E.g. <a href="https://github.com/pytorch/examples/blob/3970e068c7f18d2d54db2afee6ddd81ef3f93c24/imagenet/main.py#L177-L197">here</a> a checkpoint is loaded and the training is resumed while <a href="https://github.com/pytorch/examples/blob/3970e068c7f18d2d54db2afee6ddd81ef3f93c24/imagenet/main.py#L250-L262">here</a> the checkpoint giving the best validation accuracy is stored.</p>

Najeh_Nafti: <p>I am working with a GAN model, so I don’t calculate the accuracy, should I save the best validation G_loss and D_loss?<br>
I am using the below code during training, is it correct:</p>
<pre><code class="lang-auto">if  min_valid_loss_g &gt; valid_loss_g :
     print(f'G_Val_Loss_Decreased({min_valid_loss_g:.6f}---&gt;{valid_loss_g:.6f})\t Saving The Model')
     min_valid_loss_g = valid_loss_g
     torch.save({
            'epoch': epoch,
            'G_state_dict': G.state_dict(),
            'G_optimizer_state_dict': optimizer_G.state_dict()
            'G_loss': valid_loss_g
            },f"./generator-epoch-{epoch}.pth")
</code></pre>

ptrblck: <p>Yes, your approach sounds reasonable assuming the validation loss properly represents the training progress of your model.</p>

Najeh_Nafti: <p>How can I assume that the validation loss properly represents the training progress of my model?</p>
<p>The below code, is how I am uploading the saved models to continue training after restart the Kernel, is it correct?</p>
<pre><code class="lang-auto">G = Generator().to(device)
checkpoint = torch.load('generator.pth')
try:
    checkpoint.eval()
except AttributeError as error:
    print (error)


G.load_state_dict(checkpoint['G_state_dict'])
optimizer_G.load_state_dict(checkpoint['optimizer_state_dict_G'])
epoch = checkpoint['epoch']
loss = checkpoint['valid_loss_g']
G.eval()
</code></pre>

Najeh_Nafti: <p>should I continue training on the same training dataset or should I modify it?</p>

ptrblck: <p>The choice of the dataset depends on your use case. If you want to “continue” the training, then using the same dataset would work; if you want to fine-tune the model on another dataset then you would need to change it.</p>

Najeh_Nafti: <p>and how to load the saved models?</p>

Najeh_Nafti: <p>Using this technique, during the training I got the same loss for different epochs.<br>
Where is the problem?</p>

ptrblck: <aside class="quote no-group" data-username="Najeh_Nafti" data-post="8" data-topic="135869" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/najeh_nafti/48/36465_2.png" class="avatar"> Najeh_Nafti:</div>
<blockquote>
<p>and how to load the saved models?</p>
</blockquote>
</aside>
<p>In your code snippet you are already loading the <code>state_dict</code>s in:</p>
<pre data-code-wrap="python"><code class="lang-python">G.load_state_dict(checkpoint['G_state_dict'])
optimizer_G.load_state_dict(checkpoint['optimizer_state_dict_G'])
</code></pre>
<aside class="quote no-group" data-username="Najeh_Nafti" data-post="10" data-topic="135869" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/najeh_nafti/48/36465_2.png" class="avatar"> Najeh_Nafti:</div>
<blockquote>
<p>Using this technique, during the training I got the same loss for different epochs.<br>
Where is the problem?</p>
</blockquote>
</aside>
<p>Check that gradients are calculated for each used parameter after the first backward pass. If some <code>.grad</code> attributes are set to <code>None</code>, your computation graph is detached. If that’s not the case, try to overfit a small dataset by playing around with hyperparameters.</p>

Najeh_Nafti: <p>How can I check those gradients?</p>

ptrblck: <p>You could iterate the parameters and print their <code>.grad</code> attribute:</p>
<pre><code class="lang-python">loss.backward()
for name, param in model.named_parameters():
    print('{}, {}'.format(name, param.grad))
</code></pre>