Overfitting CNN LSTM (Time Series Prediction)

mr_cell: <p>Hi all! I’m doing Time Series Prediction with the CNN-LSTM model, but I got overfitting condition. Here is my model code:</p>
<pre><code class="lang-auto">class LSTM(nn.Module):

    def __init__(self, num_classes, input_size, hidden_size, num_layers, seq_length):
        super(LSTM, self).__init__()
        self.num_classes = num_classes
        self.num_layers = num_layers
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.seq_length = seq_length
        self.conv = nn.Conv1d(in_channels=num_classes, out_channels=num_classes, 
                              kernel_size=(3,), padding=2, 
                              padding_mode='replicate')
        self.maxpool = nn.MaxPool1d(kernel_size=2, stride=1)
        self.lstm = nn.LSTM(input_size=input_size, hidden_size=hidden_size,
                            num_layers=num_layers, batch_first=True, dropout=0.25)
        self.batchnorm = nn.BatchNorm1d(hidden_size)
        self.fc = nn.Linear(hidden_size, out_features=num_classes)
        self.sig = nn.Sigmoid()

    def forward(self, x):
        x = x.permute(0,2,1)
        conv_out = self.conv(x)
        maxpool_out = self.maxpool(conv_out)
        maxpool_out = maxpool_out.permute(0,2,1)
        h_0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size, requires_grad=True)
        c_0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size, requires_grad=True)

        output_out, (h_out, c_out) = self.lstm(maxpool_out, (h_0, c_0))

        out = output_out[:, -1, :]
        batchnorm_output = self.batchnorm(out)
        linear_output = self.fc(batchnorm_output)
        sigmoid_output = self.sig(linear_output)

        return sigmoid_output

</code></pre>
<p>And here is my train code:</p>
<pre><code class="lang-auto">seed = 23
torch.manual_seed(seed)
np.random.seed(seed)

num_epochs = 1000
learning_rate = 0.1
momentum = 0.9

input_size = 11
hidden_size = 32
num_layers = 2

num_classes = 11

model = LSTM(num_classes, input_size, hidden_size, num_layers, seq)

loss_func = nn.L1Loss()
print("-" * 70)
validator = True
optim_input = ""
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

scheduler = ReduceLROnPlateau(optimizer, mode='min', patience=75, verbose=True)

loss_list = []
epochs = []
time_per_epoch = []
# Pelatihan model
print("-" * 70)
print("Begin training with Convolutional LSTM layer (with Batch Normalization) and Adam optimizer\n")
model.train()
for epoch in range(num_epochs):
    start = time.time()
    outputs = model(train_x)
    optimizer.zero_grad()
    loss = loss_func(outputs, train_y)
    loss.backward()
    scheduler.step(loss)
    optimizer.step()
    stop = time.time()
    epoch_time = stop - start
    loss_list.append(loss.item())
    epochs.append(epoch)
    time_per_epoch.append(epoch_time)
    print("Epoch: %d, MAE: %1.5f Time: %1.3f s" % (epoch, loss.item(), epoch_time))

print("-" * 70)
print(model)

print("-" * 70)
print("Training done in %1.5f seconds with last training MAE: %1.5f" % (sum(time_per_epoch), loss_list[-1]))
plt.figure(figsize=(10, 4))
plot_function(time_step=epochs, series=loss_list,
              title='Training Loss Convolutional LSTM Layer with Batch Normalization\nOptimizer: Adam',
              xlabel='Time', ylabel='Loss', label=["loss graph"])
</code></pre>
<p>For further information, my train set shape is [764,30,11] that consist of [batch_size,sequence_length,input_size] and my test set shape is [191,30,11]. My dataset is a normalized dataset with MinMaxScaler from scikit-learn. In training, I got an MAE of 0.01567, while in testing, I got an MAE of 0.16626, it’s a very huge difference between the normalized train and test.<br>
Here is the image of the test set:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/d/3d8dc8f1698dfb2c267aa83823a7841a6c01454c.png" data-download-href="https://discuss.pytorch.org/uploads/default/3d8dc8f1698dfb2c267aa83823a7841a6c01454c" title="Plot test set"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/d/3d8dc8f1698dfb2c267aa83823a7841a6c01454c_2_690x288.png" alt="Plot test set" data-base62-sha1="8MwTHTHJNnDYn5WRX2uqbahj992" width="690" height="288" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/d/3d8dc8f1698dfb2c267aa83823a7841a6c01454c_2_690x288.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/d/3d8dc8f1698dfb2c267aa83823a7841a6c01454c_2_1035x432.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/d/3d8dc8f1698dfb2c267aa83823a7841a6c01454c.png 2x" data-dominant-color="F6F6F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Plot test set</span><span class="informations">1153×482 106 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>And here is the image of the prediction of the test set:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/7/d766e96a9aacab047b50ea734e85998618a68b1f.png" data-download-href="https://discuss.pytorch.org/uploads/default/d766e96a9aacab047b50ea734e85998618a68b1f" title="Plot prediction of test set"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/7/d766e96a9aacab047b50ea734e85998618a68b1f_2_690x297.png" alt="Plot prediction of test set" data-base62-sha1="uJx8U5KnnJlXmdX19cNmyXNHzMr" width="690" height="297" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/7/d766e96a9aacab047b50ea734e85998618a68b1f_2_690x297.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/7/d766e96a9aacab047b50ea734e85998618a68b1f_2_1035x445.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/7/d766e96a9aacab047b50ea734e85998618a68b1f.png 2x" data-dominant-color="F6F6F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Plot prediction of test set</span><span class="informations">1153×497 140 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>What am I supposed to do to reduce this overfitting condition?<br>
Thank you in advance!</p>

Billy: <p><a class="mention" href="/u/mr_cell">@mr_cell</a>, Hi.<br>
I’ve read your code and looked at your snapshots. Unfortunately I can’t read the langauge in which the snapshots labels were written but here are some pointers that work for me in CNN-LSTM Models.</p>
<p>-Increase your hidden size to something a little larger, like 128.<br>
-Decrease your number of Epochs &lt;&lt;-Definitely!.<br>
-Train your model on a larger dataset if you have one. If not, find data that is <em>almost</em> similar, something with the same input shape, and is a related task to what you want to achieve. Train your model on that data then use transfer learnining to finetune it to your original data. This will allow the model to learn the complex stuff from the other data then just figure out how to work with your new data. Basically it gives you better results from small data.</p>
<ul>
<li>Tell me how it goes. Kuddos.</li>
</ul>