Spike occurs on Loss after returning from a checkpoint

mfatih7: <p>Hello</p>
<p>While returning to training from a checkpoint spikes on training loss occurs as shown in the figure below<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/a/aaaf7bfc32645b806217c1be72252d33938e9902.png" data-download-href="https://discuss.pytorch.org/uploads/default/aaaf7bfc32645b806217c1be72252d33938e9902" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/a/aaaf7bfc32645b806217c1be72252d33938e9902_2_187x500.png" alt="image" data-base62-sha1="olX8TccAklEoG2sGi1PqpFvBCmK" width="187" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/a/aaaf7bfc32645b806217c1be72252d33938e9902_2_187x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/a/aaaf7bfc32645b806217c1be72252d33938e9902_2_280x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/a/aaaf7bfc32645b806217c1be72252d33938e9902.png 2x" data-dominant-color="F1EEF2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">335×894 37.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>While defining loss, optimizer and learning rate scheduler I use</p>
<pre><code>criterion=torch.nn.MSELoss(size_average=True, reduce=True, reduction='mean')

optimizer=torch.optim.Adam(model.parameters(), lr=learning_rate)

lr_scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(
                                                               optimizer,
                                                               mode='min',
                                                               factor=0.75,
                                                               patience=math.ceil(patience_chunk_count),
                                                               threshold=0.0001,
                                                               threshold_mode='rel',
                                                               cooldown=0,
                                                               min_lr=0,
                                                               eps=1e-08,
                                                               verbose=True,
                                                          ) 
</code></pre>
<p>While saving checkpoint I use</p>
<pre><code>    checkpoint = {
                 ...                      
                  'model_state_dict': model.state_dict(),
                  'optimizer_state_dict': optimizer.state_dict(),
                  'lr_scheduler_state_dict': lr_scheduler.state_dict(),                      
                  ...
                 }
</code></pre>
<p>While loading from a checkpoint I use</p>
<pre><code>checkpoint = torch.load(path_checkpoints_file, map_location=torch.device('cpu'))

model.load_state_dict(checkpoint['model_state_dict'])    
model.to(device)    
optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
lr_scheduler.load_state_dict(checkpoint['lr_scheduler_state_dict'])
</code></pre>
<p>And my training loop is something like below</p>
<pre><code>for epoch in range (currentEpoch, num_epochs+1):
        ...
        learning_rate = optimizer.param_groups[0]['lr']
        ...
        model.train()            
        
        for i,(inputs, labels, inputs_metadata) in enumerate(train_loader):
            
            inputs = inputs.to(device)
            labels = labels.to(device)
        
            outputs = model(inputs)  
            
            loss_train=criterion(outputs, labels)
            
            optimizer.zero_grad()
            loss_train.backward()
            optimizer.step()
            ... 
</code></pre>
<p>I see other people struggling with the same problem.<br>
Does anyone know the reason for the problem?<br>
How can I solve the problem in a naive way?</p>

mfatih7: <p>Hello</p>
<p>The reason for the spike was because of randomization of training data after a restart to training.</p>
<p>Nothing is wrong with Pytorch functions.</p>
<p>best regards</p>