Abnormal Loss curve after cosing annealing

Rohan_Kumar: <p>I used cosine annealing on my transformer network  and got this curve<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/9/b93ef33df4052c66ac32bc767a9f2980770cdab0.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/b93ef33df4052c66ac32bc767a9f2980770cdab0" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/9/b93ef33df4052c66ac32bc767a9f2980770cdab0_2_690x364.jpeg" alt="image" data-base62-sha1="qqLcx139q64NbvXAUIUiGHJBjQ4" width="690" height="364" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/9/b93ef33df4052c66ac32bc767a9f2980770cdab0_2_690x364.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/b/9/b93ef33df4052c66ac32bc767a9f2980770cdab0_2_1035x546.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/9/b93ef33df4052c66ac32bc767a9f2980770cdab0.jpeg 2x" data-dominant-color="F9FAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1191×630 68.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I have used Cross Entropy as my loss function and  this is my parameters for the scheduler  with max lr = 1e-3</p>
<pre><code class="lang-auto">sch = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, len(train_data) , eta_min=0, last_epoch=-1)
</code></pre>
<p>And this is my training code, what can i do to improve this loss curve?</p>
<pre><code class="lang-auto">for epoch in tqdm(range(epochs)):
        model.train()
        epoch_loss = 0
        for batch in tqdm(train_iterator):
            src_inp = batch['src_sentence'].permute(1,0).cuda()
            trg_inp = batch['trg_sentence'].permute(1,0).cuda()
            in_tgt = trg_inp[:-1, :]
            exp_tgt = trg_inp[1:, :]
            optimizer.zero_grad()
            output= model(src_inp, in_tgt)

            output = output.view(-1, output.shape[-1])

            trg = exp_tgt.reshape(-1)
            
            loss = criterion1(output, trg)
            loss.backward()

            torch.nn.utils.clip_grad_norm_(model.parameters(), 0.25)
            for p in model.parameters():
                p.data.add_(-lr, p.grad)
            optimizer.step()
            sch.step()
            epoch_loss += loss.item()
        y.append(epoch_loss/len(train_iterator))
        epoch_loss = 0 
        model.eval()
        for batch in tqdm(val_iterator):
            src_inp = batch['src_sentence'].permute(1,0).cuda()
            trg_inp = batch['trg_sentence'].permute(1,0).cuda()
            in_tgt = trg_inp[:-1, :]
            exp_tgt = trg_inp[1:, :]
            optimizer.zero_grad()
            output= model(src_inp, in_tgt)

            output = output.view(-1, output.shape[-1])

            trg = exp_tgt.reshape(-1)

            loss = criterion1(output, trg)

            epoch_loss += loss.item()
            if(math.isnan(loss.item())):
                print(src_inp)
        y_val.append(epoch_loss/len(val_iterator))

</code></pre>

Kushaj: <p>Maybe batch size and validation set size is small.</p>