Model still highly overfitting

Username2: <p>I’m trying to <strong>forecast the product demand of several articles</strong>. I’m using a CNN-LSTM model and work with continuous and categorical features. (I embed the categorical data in the model).<br>
I have daily data (a time series from 2013 until now) for around 100 article groups.<br>
I only try to predict the demand of articles that get bought often. (and drop the data for the articles I have less data for)</p>
<p>At the start I summarized the data by month and used the last 12 months to predict the product demand for 1-6 months in the future.<br>
That worked fine for the training data (around 90-99% accuracy), but I only got around 10-15 % accuracy for the test data. (I split the data: 85% train and 15% test) So the model was overfitting a lot. I added <strong>drop out layers</strong> and used <strong>weight decay</strong> and <strong>early stopping</strong>, but nothing worked.</p>
<p>Then, to get “more” data, I <strong>summarized the data on a weekly basis</strong> and now use the last 52 weeks o predict the demand of the next week (and then use the "new"data to predict the demand for another week in the future, and so on, step by step).<br>
I am using a <strong>SGD optimizer</strong>. For the monthly data I used a CosineAnnealingWithWarmRestarts Scheduler, for the weekly data I first switched to an circular/triangular decreasing scheduler and now I am using the <strong>OneCycle scheduler</strong> (with the help of a learning rate finder: LRFinder).<br>
To find better hyperparameters for the number of layers, filters, drop out probabilities, weight decay, base momentum, etc.) I am using Optuna. The metric to decide which trial is “better” than another is the test accuracy. (To also compare <strong>simpler and more complex models</strong> regarding the number of neurons ← battling the overfitting).<br>
I run a lot of trials with <strong>different hyperparameters combinations</strong> now, but the model is <strong>still overfitting (a lot!)</strong> and the train accuracy is no longer as high as with the monthly data.</p>
<p>Any ideas what I could do or improve?</p>
<p>My model:</p>
<pre><code class="lang-auto">CNNLSTM(
  (emb_layers): ModuleList(
    (0): Embedding(8, 4)
    (1): Embedding(12, 6)
    (2): Embedding(53, 27)
    (3): Embedding(2, 1)
    (4): Embedding(2, 1)
    (5): Embedding(2, 1)
    (6): Embedding(2, 1)
    (7): Embedding(28, 14)
    (8): Embedding(4, 2)
  )
  (emb_dropout_layer): Dropout(p=0.386, inplace=False)
  (cnn): CNN(
    (conv1): Conv2d(52, 256, kernel_size=(5, 5), stride=(1, 1))
    (conv2): Conv2d(256, 128, kernel_size=(3, 3), stride=(1, 1))
    (maxpool): MaxPool2d(kernel_size=(3, 3), stride=1, padding=0, dilation=1, ceil_mode=False)
    (drop): Dropout(p=0.1599, inplace=False)
  )
  (lstm): LSTM(85, 96, num_layers=2, batch_first=True, dropout=0.281)
  (linear1): Linear(in_features=96, out_features=28, bias=True)
  (linear2): Linear(in_features=1, out_features=1, bias=True)
  (lbefore_lin_drop): Dropout2d(p=0.153, inplace=False)
  (fin_drop): Dropout(p=0.073, inplace=False)
  (leakyrelu): LeakyReLU(negative_slope=0.15)
)
</code></pre>
<p>So I embed the categorical features first, then concatenate them and pass it to a drop out layer.<br>
Then I concatenate the embedded data with the continuous data and pass it to a CNN that consists of 2 convolutional layers Conv2D followed by a MaxPool2d and a Dropout layer.<br>
<em>The input for the CNN is of the shape (Number of samples x Number of articles x Time span x number of features), for weekly data the time span is 52 weeks, the number of features is the sum of the embedding dimensions plus the number of continuous features.</em><br>
Then a LeakyReLU.<br>
Then I reshape the data before i pass them to the LSTM.<br>
<em>(I reshape the data from (N x out_channel x H_out x W_out) to (N</em> * <em>out_channel x H_out x W_out).)</em><br>
Then a Dropout2d layer.<br>
Then a Linear layer.<br>
More reshaping.<br>
<em>(I reshape the data from (N x number of articles</em> * <em>length of the forecast) to (N x number of articles x length of the forecast)), at the moment the length of forecast is 1 week, because I predict step by step, I also tried multiple step forecast before and the length of forecast was 12 weeks.)</em><br>
Another LeakyReLU.<br>
Then a final Linear layer followed by another Dropout layer and a LeakyReLU.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/e/3ea49fc1f82927c7f2625ef993bfc047fc2d199f.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/3ea49fc1f82927c7f2625ef993bfc047fc2d199f" title="newplot (2)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/e/3ea49fc1f82927c7f2625ef993bfc047fc2d199f_2_517x90.jpeg" alt="newplot (2)" data-base62-sha1="8Waj8afolMQK8Yw7hHWPqL1gTkX" width="517" height="90" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/e/3ea49fc1f82927c7f2625ef993bfc047fc2d199f_2_517x90.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/3/e/3ea49fc1f82927c7f2625ef993bfc047fc2d199f_2_775x135.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/e/3ea49fc1f82927c7f2625ef993bfc047fc2d199f_2_1034x180.jpeg 2x" data-dominant-color="E8ECF1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">newplot (2)</span><span class="informations">3017×525 293 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
This is an overview over some trials, as you can see the test accuracy gets nowhere fast…</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/c/ec0dcf595d968f0ef384f343dc9e41619c606077.png" data-download-href="https://discuss.pytorch.org/uploads/default/ec0dcf595d968f0ef384f343dc9e41619c606077" title="Loss2"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/c/ec0dcf595d968f0ef384f343dc9e41619c606077_2_345x240.png" alt="Loss2" data-base62-sha1="xGegMEUrgNXJzeJQAaSLPAy4Tob" width="345" height="240" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/c/ec0dcf595d968f0ef384f343dc9e41619c606077_2_345x240.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/c/ec0dcf595d968f0ef384f343dc9e41619c606077_2_517x360.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/c/ec0dcf595d968f0ef384f343dc9e41619c606077_2_690x480.png 2x" data-dominant-color="F8F4EF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Loss2</span><span class="informations">712×496 69 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/2/d2f1028486084605a26f1dbe572f0524ab508457.png" data-download-href="https://discuss.pytorch.org/uploads/default/d2f1028486084605a26f1dbe572f0524ab508457" title="Accuracy2"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d2f1028486084605a26f1dbe572f0524ab508457_2_345x240.png" alt="Accuracy2" data-base62-sha1="u64D3EJpL3Vfi94na3MslqaQgcL" width="345" height="240" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d2f1028486084605a26f1dbe572f0524ab508457_2_345x240.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d2f1028486084605a26f1dbe572f0524ab508457_2_517x360.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d2f1028486084605a26f1dbe572f0524ab508457_2_690x480.png 2x" data-dominant-color="F8F4F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Accuracy2</span><span class="informations">713×496 61.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Just an example for a loss and accuracy history I get.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/0/b0d9f1cc1b39929aada012a406db02dadfe64d24.png" data-download-href="https://discuss.pytorch.org/uploads/default/b0d9f1cc1b39929aada012a406db02dadfe64d24" title="Download (3) (1)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/0/b0d9f1cc1b39929aada012a406db02dadfe64d24_2_393x500.png" alt="Download (3) (1)" data-base62-sha1="peuYGqVtmpE9OFrvzhvX5t1g4i8" width="393" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/0/b0d9f1cc1b39929aada012a406db02dadfe64d24_2_393x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/0/b0d9f1cc1b39929aada012a406db02dadfe64d24_2_589x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/0/b0d9f1cc1b39929aada012a406db02dadfe64d24_2_786x1000.png 2x" data-dominant-color="EEEFF4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Download (3) (1)</span><span class="informations">1148×1459 324 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
And the predictions look like this at the moment…</p>

Username2: <p>I reduced the number of features (from 45 to 36) and dropped the last linear layer, but still no improvement regarding the overfit.</p>
<p>What I am doing wrong?</p>