Training A Forecasting Model With CNN's and LSTM's

username: Passisimple
cooked: <p>I have been working on a prediction problem for quite some time.</p>
<p>My model has the input shape of [Sequence Length , Attributes] and tries to predict a single attribute at next time step.</p>
<p>I used very similar model before and got a good result on a very similar problem. Also, I am trying to keep my model as computationally cheap as possible to use in some part of my thesis.</p>
<p>Some of the attributes have critical outliers.</p>
<p><strong>Aim:</strong> Predict Next Time Steps Attribute, append it to the current sequential data and keep predicting. Removing the oldest time step,and keeping sequence length at a fixed length</p>
<p><strong>My Model Description:</strong> 5-6 CNN Layers 1 LSTM Layer 1 FC Layer (With ReLU Activations Except LSTM of course.)(This is just to show model idea. Layer Count Type etc. can be changed.)<br>
I am able to choose, sequential length of the input data</p>
<p><strong>What I Tried:</strong></p>
<ol>
<li>Increased &amp; Decreased CNN Layers</li>
<li>Changing Filter Sizes.</li>
<li>Added Batch Normalization Layers between CNN’s</li>
<li>Added Max Pooling, Average Pooling &amp; Striding On a Sequence</li>
<li>Increased &amp; Decreased Channel Sizes for CNN Layers</li>
<li>Used GRU’s , LSTM’s and RNN’s for sequential Part</li>
<li>Increased &amp; Decreased Hidden Dimension of Sequential Layer</li>
<li>Tried multiple Sequential Layers</li>
<li>Tried Variety of Sequence Lengths</li>
<li>Added Multiple FC Layer’s at the end of Sequential Layer</li>
<li>Tried Transforming the data, “Yeo-Johnson”, Logarithmic, Cubic etc.</li>
<li>Tried Only Sequential Layer-Based &amp; Only CNN Based Models</li>
<li>Different Batch Size, Much Larger Training Set , Different Number of Epochs…</li>
</ol>
<p>Forecasting made with model:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/0/2004c5c271cd09eaf62de41363defc4d80da2ce7.png" data-download-href="https://discuss.pytorch.org/uploads/default/2004c5c271cd09eaf62de41363defc4d80da2ce7" title="Capture"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/0/2004c5c271cd09eaf62de41363defc4d80da2ce7_2_690x430.png" alt="Capture" data-base62-sha1="4zfvfHuIoHauOAHsxH5mlNJIMeP" width="690" height="430" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/0/2004c5c271cd09eaf62de41363defc4d80da2ce7_2_690x430.png, https://discuss.pytorch.org/uploads/default/original/3X/2/0/2004c5c271cd09eaf62de41363defc4d80da2ce7.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/0/2004c5c271cd09eaf62de41363defc4d80da2ce7.png 2x" data-dominant-color="F9F7F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture</span><span class="informations">853×532 37.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I kind of ran out of ideas at this point. Since, I am not very experienced in DL, but a long review of multiple sources, didn’t help me at this point. What seems to be the problem here ? How can I approach this model ? What are the fundamental methods I can try here ?</p>
<p>Thank you for your time.</p>

username: jacksth22
cooked: <p>Can you provide the code and a small sample of the data? It’s hard to know whether there is an error in your code or you are using an inappropriate model.</p>
<p><a href="https://pypi.org/project/tsai/" rel="noopener nofollow ugc">TSAI</a> may be worth looking into. Also, I recently had surprising success with an extremely simple, one-layer LSTM for multivariate time-series prediction (validation MAPE reached &lt;1%, beating 10+ other architectures I tried). It may be worth following <a href="https://keras.io/examples/timeseries/timeseries_weather_forecasting/" rel="noopener nofollow ugc">this example</a> on your dataset.</p>