Resume training from checkpoints produces different losses

username: ilyes
cooked: <p>Hello,</p>
<p>So as the title states, I am having peaks in the loss when I resume training eventhough I am saving everything in the checkpoint : model state, optimizer state, and having a manual seed. like indicated below.</p>
<p><strong>Dataloaders:</strong><br>
a function that returns the dataloaders at the start of my training program.</p>
<pre><code class="lang-auto">    torch.manual_seed(1)
    indices = torch.randperm(len(train_dataset)).tolist()
    train_idx, valid_idx, test_idx = indices[val_size + test_size:], indices[:val_size], indices[
                                                                                         val_size:test_size + val_size]
    train_sampler = SubsetRandomSampler(train_idx)
    val_sampler = SubsetRandomSampler(valid_idx)
    test_sampler = SubsetRandomSampler(test_idx)

    train_loader = DataLoader(train_dataset, batch_size=batch_size, sampler=train_sampler,
                              pin_memory=torch.cuda.is_available(), num_workers=4)
    val_loader = DataLoader(val_dataset, batch_size=batch_size, sampler=val_sampler,
                            pin_memory=torch.cuda.is_available(), num_workers=4)
    test_loader = DataLoader(val_dataset, batch_size=batch_size, sampler=test_sampler,
                             pin_memory=torch.cuda.is_available(), num_workers=4)
</code></pre>
<p><strong>Saving:</strong><br>
I save the checkpoint just after the training pass, and before doing validation or test, means the model is still model.train().</p>
<pre><code class="lang-auto">        torch.save({
            'epoch': e,
            'model_state_dict': mymodel.state_dict(),
            'best_loss': best_loss,
            'optimizer_state_dict': optim.state_dict(),
        }, os.path.join("Checkpoints", path, 'training_state.pt'))
</code></pre>
<p><strong>Loading:</strong><br>
I check if there is a checkpoint at a given directory, and load it at the start of my training program.</p>
<pre><code class="lang-auto">    starting_epoch = 0
    best_loss = 100000
    if os.path.exists(os.path.join("Checkpoints", path, 'training_state.pt')):
        checkpoint = torch.load(os.path.join("Checkpoints", path, 'training_state.pt'), map_location=device)
        mymodel.load_state_dict(checkpoint['model_state_dict'])
        optim.load_state_dict(checkpoint['optimizer_state_dict'])
        starting_epoch = checkpoint['epoch']
        best_loss = checkpoint['best_loss']
</code></pre>
<p>Do you guys see anything that may cause peaks in the loss after resuming training ? it goes for example from 0.54 to 0.7 when resumed</p>

username: ptrblck
cooked: <p>The posted code looks fine.<br>
Could you post some information about your model and training routine?<br>
If possible, could you post your model definition?</p>

username: pinocchio
cooked: <p>did you ever solve this issue?</p>
<hr>
<p>related:</p>
<aside class="quote quote-modified" data-post="1" data-topic="60683">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/94ad74/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/resume-iterating-dataloader-from-checkpoint-batch-idx/60683">Resume iterating dataloader from checkpoint batch_idx</a> 
  </div>
  <blockquote>
    Hi, 
I was wondering whether it is possible to resume iterating through a dataloader from a checkpoint. 
For example: 
    dataloaders_dict = {phase: torch.utils.data.DataLoader(datasets_dict[phase], batch_size=args.batch_size, num_workers=args.num_workers, shuffle=False) for phase in ['train']} # make sure shuffling is false incase you restart

if os.path.isdir(args.batchidx_checkpoint):
            checkpoint = torch.load(args.batchidx_checkpoint + 'batches.pt')
            batch_idx = checkpo…
  </blockquote>
</aside>

<aside class="quote quote-modified" data-post="1" data-topic="33053">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/90db22/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/resuming-from-model-checkpoints-produces-different-training-loss/33053">Resuming from model checkpoints produces different training loss</a> 
  </div>
  <blockquote>
    Hello, 
for the last 2 days I am trying to solve issue when resuming training from model checkpoint. Problem is that the training loss after resuming is a LOT different than before saving model (the difference is huge, almost as if the model was right after initialization process). I can see, that after few iterations it increases accuracy (decreases loss) much faster than if it was learned from scratch. But I don’t know what cuases the loss to be so high. 
But weird thing is that I checked outp…
  </blockquote>
</aside>


username: pranayKD
cooked: <p>Hi,</p>
<p>I am facing the exact same problem. Is there any solution for the training to continue from the same loss?</p>
<p>Thanks</p>

username: blenderender
cooked: <p>Does the peak in loss occur when using adaptive optimizers or even with something like SGD with no momentum?</p>

username: pranayKD
cooked: <p>I am doing some experiments to figure out the root cause. Its happening even in case of SGD optmizer with no momentum.</p>
<p><code>optimizer = optim.SGD(net.parameters(), lr=0.001)</code></p>
<p>In the following image, the slight loss jump after 2000 iterations, is after the training is continued from a checkpoint.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/a/1/a1e6d6a7723ec466effb5b19378cc64c036b7e33.png" alt="image" data-base62-sha1="n6froMFWcafGsesH14m6r3EyHvR" width="378" height="264"></p>

username: pranayKD
cooked: <p>Found the root cause for my problem. Nothing to do with model saving or loading. Just a silly mistake.</p>
<aside class="quote quote-modified" data-post="6" data-topic="88373">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pranaykd/48/26474_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/unable-to-pause-and-resume-training-loading-a-model-without-getting-a-small-jump-in-training-loss/88373/6">Unable to pause and resume training (loading a model) without getting a small jump in training loss</a> 
  </div>
  <blockquote>
    After spending few hours trying to figure out the problem, found the root cause. And it has nothing to do with model / optimizer loading and saving at all. 
What I needed to change is 
lab_to_val = {j:i for i,j in enumerate(training_classes)}

To 
lab_to_val = {j:i for i,j in enumerate(sorted(training_classes))}

Since the label to value conversion was not happening on the sorted list, each time my runtime was reset, a single label would take different values. Ex - if person class takes value 1 …
  </blockquote>
</aside>