Loss goes from 0.00xx to xxxx.xx

Diaa_elsayed_ziada: <p>I’m working on 3d landmark detection model<br>
the training process works very well for el first 2 epochs and at the middle of the epoch 3 the loss starts to increase in an insane way and model results get much worth than it’s previous<br>
what could be the reason of this</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/6/96eb4c96e46a138c989d6807196be3469166a52b.png" data-download-href="https://discuss.pytorch.org/uploads/default/96eb4c96e46a138c989d6807196be3469166a52b" title="Screenshot from 2020-03-30 19-56-25"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/6/96eb4c96e46a138c989d6807196be3469166a52b.png" alt="Screenshot from 2020-03-30 19-56-25" data-base62-sha1="lx5KiqFqqo08WWLUCjURsQkLa9d" width="517" height="142" data-dominant-color="E6E6E6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2020-03-30 19-56-25</span><span class="informations">831×230 12.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre><code class="lang-auto">def train(train_loader, model, optimizer, criterion, controller):
    model.train()
    t = time()
    train_loss = controller.current_train_loss
    for i, data in enumerate(train_loader):
        images = data['image']
        if images.shape[1] == 0:
            print(f"\rTaining finished batches {i+1}/{len(train_loader)}   {int(((i+1)/len(train_loader))*100)}%", end='')
            continue
        targets = data['keypoints']
        targets = targets.view(targets.size(0), -1)
        
        images = images.type(torch.FloatTensor)
        targets = targets.type(torch.FloatTensor)
        
        images = images.to(device)
        targets = targets.to(device)
        
        optimizer.zero_grad()
        
        output = model(images)#[0] for 2d model
        loss = criterion(output, targets)
        loss.backward()
        optimizer.step()
        
        train_loss += 1/(i + 1) * (loss.data - train_loss)
        
        # print(targets.shape, output.shape)

        del images
        del targets
        torch.cuda.empty_cache()
        
        delay = time() - t

        print(f"\rTaining finished batches {i+1}/{len(train_loader)}   {int(((i+1)/len(train_loader))*100)}%   delay {int(delay)}s   time left {int(delay* (len(train_loader)-(i+1)))}s   loss { loss.item()}",end='\n')
        if i%10 == 0:
            controller.update_batch_info(i, model, optimizer, train_loss)
        t = time()
        
    return train_loss
</code></pre>

albanD: <p>Hi,</p>
<p>I am not a specialist but the usual reasons I have seen are:</p>
<ul>
<li>learning rate too high</li>
<li>loss function is non-smooth for low values</li>
</ul>
<p>This is most likely the second one here if it behaved properly at the beginning of the training.<br>
You want to make sure that small loss value don’t make anything numerically unstable.</p>

Diaa_elsayed_ziada: <p>could tell me more about (loss function is non-smooth for low values)<br>
how to deal with to continue my training<br>
i’m using mse loss</p>

albanD: <p>If the loss itself is MSE, it should be fine.<br>
Do you have special layers in your net?</p>
<p>The way to continue training is just to make sure you don’t have these instabilities.</p>

Diaa_elsayed_ziada: <p>no special layers only a deep CNN network<br>
and i worked well at first 3 epochs and this happens after 47% of the epoch 4</p>
<p>i reduced learning rate from 0.001 to 0.0001, and eps from 1e-8 to ie-7 and restart training at epoch 4<br>
it’s working fine until now<br>
if it start to do it again i will follow up with you<br>
thanks for following up</p>