Is there some problems use resume in Pytorch?

username: CSUMIT
cooked: <p>When i train my network, in 156th epochs the training is break. So i use <code>--resume</code> to continue training(load the last checkpoint).<br>
But i see one phenomenon: the train loss will rise from 0.0870 to 0.1321, and it will take many epochs decrease to 0.08 again.</p>
<p>I also meet this problem in other training(after use <code>--resume</code> to load last checkpoint, the training loss will raise from 0.1557 to 0.1999. The val loss will raise from 0.2488 to 0.2657 and the iou will decrease from 0.6419 to 0.6009.):<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/e/ed293f416eb4f9be31719436cade7f97c3399e6b.jpg" data-download-href="https://discuss.pytorch.org/uploads/default/ed293f416eb4f9be31719436cade7f97c3399e6b" title="2018-04-12_142748"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/e/ed293f416eb4f9be31719436cade7f97c3399e6b_2_690x476.jpg" alt="2018-04-12_142748" data-base62-sha1="xQ1wYWDT3kzNVPGCUDhffeqcd1V" width="690" height="476" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/e/ed293f416eb4f9be31719436cade7f97c3399e6b_2_690x476.jpg, https://discuss.pytorch.org/uploads/default/original/2X/e/ed293f416eb4f9be31719436cade7f97c3399e6b.jpg 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/e/ed293f416eb4f9be31719436cade7f97c3399e6b.jpg 2x" data-dominant-color="D1E3F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">2018-04-12_142748</span><span class="informations">813×562 145 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
So use <code>--resume</code> to continue the training will reduse accuracy? Is it normal in this case?<br>
Thank you.</p>

username: ptrblck
cooked: <p>It depends how you’ve implemented your “resume” logic. From your description I assume you are just loading the <code>state_dict</code> and start the training with a new optimizer.<br>
Using an “adaptive” optimizer might worsen your accuracy, since the “old” optimizer had some internal states, momentum etc., while the new one will have a cold start.<br>
You could try to save the optimizer’s <code>state_dict</code> as well.</p>
<p>However, why did the training break in the first place? Could you post the error and some information?</p>

username: CSUMIT
cooked: <p>Hi, here is the code i use resume:</p>
<blockquote>
<p>save_checkpoint({<br>
‘epoch’: epoch + 1,<br>
‘arch’: str(model),<br>
‘state_dict’: model.state_dict(),<br>
‘best_acc’: best_acc,<br>
‘optimizer’ : optimizer.state_dict(),<br>
}, is_best, filenameCheckpoint, filenameBest)</p>
</blockquote>
<blockquote>
<pre><code>if args.resume:
    if enc:
        filenameCheckpoint = savedir + '/checkpoint_enc.pth.tar'
    else:
        filenameCheckpoint = savedir + '/checkpoint.pth.tar'

    assert os.path.exists(filenameCheckpoint), "Error: resume option was used but checkpoint was not found in folder"
    checkpoint = torch.load(filenameCheckpoint)
    start_epoch = checkpoint['epoch']
    model.load_state_dict(checkpoint['state_dict'])
    optimizer.load_state_dict(checkpoint['optimizer'])
    best_acc = checkpoint['best_acc']
    print("=&gt; Loaded checkpoint at epoch {})".format(checkpoint['epoch']))
</code></pre>
</blockquote>
<p>In the code, maybe i have save the <code>state_dict</code>, so i also don’t know how to solve this problem.</p>
<p>About the training break, it is normal for me. Because i use a free machine from my school. It will break every 10 hours. I know it’s hurt, but i can’t do something for that.</p>
<p>Thank you.</p>

username: ptrblck
cooked: <p>Looks perfectly fine to me. Just like the <a href="https://github.com/pytorch/examples/blob/master/imagenet/main.py#L98">ImageNet example</a>.</p>
<p>Which optimizer are you using? I would like to check if with a small dummy example.</p>

username: CSUMIT
cooked: <p>Hi ptrblck,<br>
my optimizer is:<br>
<code>optimizer = Adam(model.parameters(), 5e-4, (0.9, 0.999), eps=1e-08, weight_decay=1e-4)</code></p>
<p>Thank you.</p>

username: ptrblck
cooked: <p>I created a small <a href="https://gist.github.com/ptrblck/a496b81ec705d4a8fe951becb2aa79c6">dummy example</a> and cannot recreate the issue.<br>
I wanted to try of layers like <code>Dropout</code> and <code>BatchNorm</code> could possibly change something, but at least in the same terminal with a seed the model returns the same loss values.</p>
<p>Still, it could be an issue with the random number generator, although I cannot explain, why the loss spikes that much.<br>
Could you try to compare the predictions after calling <code>model.eval</code>?</p>

username: CSUMIT
cooked: <p>Thank you, i will try to find other method to solve this problem.<br>
Thanks for your kindly response.<img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=5" title=":+1:" class="emoji" alt=":+1:"></p>

username: senmao
cooked: <p>Hi I met the same issue, have you figured out it?</p>
<p>Best</p>

username: 111248
cooked: <p>I have the same issue, too. After I resume the model, loss is much higher than before.</p>

username: andreys42
cooked: <p>Hello, <a class="mention" href="/u/ptrblck">@ptrblck</a>. What if I use pytorch lightning Trainer.fit to train the model, how can I resume training?<br>
My approach is:</p>
<pre><code class="lang-auto">model = FaultNetPL(batch_size = 5).cuda()

filenameCheckpoint = 'experiments/FaultNet/epoch=187-avg_valid_iou=0.6822.ckpt'
checkpoint = torch.load(filenameCheckpoint)

trainer = Trainer(checkpoint_callback=checkpoint_callback, 
                           resume_from_checkpoint = checkpoint,
                           max_epochs=400,
                           gpus=1,
                           logger=logger)

trainer.fit()
</code></pre>
<p>Is this correct approach ?</p>

username: ptrblck
cooked: <p>I don’t know, how Lightning resumes the training, so <a href="https://forums.pytorchlightning.ai/t/how-to-resume-training/432">this topic</a> might be helpful. CC <a class="mention" href="/u/williamfalcon">@williamFalcon</a></p>