Spikes in learning curve for LSTM model

dliousas: <p>I’m trying to use an LSTM model to predict the right hand side of the system of ODE’s for a pendulum.</p>
<p>Problem: There are several strange spikes in my learning curves:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/5/d53e5dcac1ba4366ceda73bdd6f67d05b664aa9a.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/d53e5dcac1ba4366ceda73bdd6f67d05b664aa9a" title="Week 11 (2)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/5/d53e5dcac1ba4366ceda73bdd6f67d05b664aa9a_2_690x388.jpeg" alt="Week 11 (2)" data-base62-sha1="uqrjSbySDlUHo1huIitMzwDLwXo" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/5/d53e5dcac1ba4366ceda73bdd6f67d05b664aa9a_2_690x388.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/d/5/d53e5dcac1ba4366ceda73bdd6f67d05b664aa9a.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/5/d53e5dcac1ba4366ceda73bdd6f67d05b664aa9a.jpeg 2x" data-dominant-color="E2E2DF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Week 11 (2)</span><span class="informations">960×540 93.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Turns out if I make <code>drop_last=True</code> in my dataloader those spikes disappear.</p>
<p>Dropping the last mini batch seems to be a workaround. Is there a better way to address this problem? Is there a bug in my training loop that causes this?<br>
Also, this appears to only happen when the sequence length is long (this one is 248*5.5=1364). Smaller sequence lengths do not have spikes like that, which also makes me think I have a bug somewhere.</p>
<pre><code class="lang-auto">def train_model(datatrain_IP, datatrain_OP, time_data, n_val, batch_size, model, nepoch, lrate, reg_factor, reg_type, device_name,
                epoch_in=0, train_loss_history=(), val_loss_history = (), drop_last=False):
    # define the loss_func &amp; optimizer
    loss_func = nn.MSELoss()
    running_loss = 0.0
    optimizer = optim.NAdam(model.parameters(), lr=lrate)
    # randomly split into train &amp; validation data loaders
    train_dl, val_dl = prepare_data(datatrain_IP, datatrain_OP, time_data, n_val, batch_size, drop_last=drop_last)
    # enumerate epochs
    for epoch in range(epoch_in,nepoch):
        model.train()
        # enumerate mini batches
        for i, (inputs, targets, _) in enumerate(train_dl):
            inputs  = inputs.to(device_name, non_blocking=False)
            targets = targets.to(device_name, non_blocking=False)
            # clear the gradients
            optimizer.zero_grad()
            # compute the model output
            yhat = model(inputs)
            # calculate loss
            loss = loss_func(yhat, targets)
            # find gradient of loss w.r.t tensors
            loss.backward()
            # update model weights
            optimizer.step()
            # calculate statistics
            running_loss += loss.item()
        # calculate average mse amongst all mini-batches in current epoch
        train_loss_history = train_loss_history + (running_loss/(i+1),)        
        running_loss = 0.0
        # Validation
        model.eval()
        # validation loss
        mse, actuals, predictions, val_time = evaluate_model(val_dl, model, device_name)
        val_loss_history = val_loss_history + (mse,)
    return train_loss_history, val_loss_history, optimizer, epoch, loss, val_data_actual, val_data_predictions, val_time
</code></pre>