Unable to achieve better performance with transformer than LSTM

Mahesha999: <p>I am trying to train an ML model on time series data. The input is 10 timeseries which are essentially a sensor data. The output is another set of three time series. I feed  the model with the window of 100. So, the input shape becomes <code>(100, 10)</code>. I want to predict output time series values for single time step. So, the output shape becomes <code>(1, 3)</code>. (If I create mini batches of size say <code>x</code>, the input and output shapes become <code>(x, 100, 10)</code> and <code>(x, 1, 3)</code>).</p>
<p>My approach is to first overfit the model on smaller number of records. See if model is indeed learning / able to overfit the data. Then add some regularization (mostly dropout) and then try to train the model on full dataset.</p>
<p>First, I tried to overfit LSTM model (4 LSTM layers with 1400 hidden units each) on small dataset and visualised the outcome. It did well. So, I tried to train it on the whole dataset. It did okayish, but still struggled at some places.</p>
<p>I tried adding dropouts too, but it did not yield any significant improvement. So, I tried to train <a href="https://huggingface.co/docs/transformers/en/model_doc/patchtst" rel="noopener nofollow ugc">PatchTST transformer model</a>. First, I tried to overfit smaller model and did well. In fact, when I visualized the output, I realised that it was able to get tighter overfit than the LSTM model. So, I tried to train it on the whole dataset.</p>
<p>The initial version of PatchTST I tried is as follows:</p>
<pre><code>config = PatchTSTConfig(
    num_input_channels=10,
    context_length=100,
    num_targets=3,
    patch_length=10,
    patch_stride=10,
    prediction_length=1,
    num_hidden_layers=4,
    num_attention_heads=8,
    d_model=600,
)

model = PatchTSTForRegression(config)

# Model Trainer body looks like this:
patch_tst_y_hat = self.model(X)
y_hat = patch_tst_y_hat.regression_outputs
# for LSTM it was:
# y_hat = self.model(X)
loss = self.loss_fn(y_hat, Y)
self.optimizer.zero_grad()   
loss.backward()
self.optimizer.step()
</code></pre>
<p>With this as base config, I tried different changes to it for hyper parameter optimization:</p>
<ol>
<li>layers = 7</li>
<li>d_model = 1000</li>
<li>d_model = 1000, num_attention_heads = 4</li>
</ol>
<p>The validation loss curves as follows:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/4/341967a54729e3d3ed761c52a70bd7b7d818be20.png" data-download-href="https://discuss.pytorch.org/uploads/default/341967a54729e3d3ed761c52a70bd7b7d818be20" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/4/341967a54729e3d3ed761c52a70bd7b7d818be20_2_690x410.png" alt="image" data-base62-sha1="7qTg0hi3JxlR30eTzAVmQhUqe9G" width="690" height="410" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/4/341967a54729e3d3ed761c52a70bd7b7d818be20_2_690x410.png, https://discuss.pytorch.org/uploads/default/original/3X/3/4/341967a54729e3d3ed761c52a70bd7b7d818be20.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/4/341967a54729e3d3ed761c52a70bd7b7d818be20.png 2x" data-dominant-color="FDFCFD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">766×456 31.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The training loss curves also look the same: <a href="https://i.postimg.cc/3wkgMshH/image.png" rel="noopener nofollow ugc">image</a> (Forum is allowing me to insert only one image)</p>
<p>Can someone suggest any idea to improve LSTM model performance further with PatchTST or some other transformer related changes?</p>
<p>PS: I have also tried LSTM with attention, it did yield tiny improvement, which is not sufficient. I am also open for any non-transformer related suggestions too. But will love to learn how can we achieve better performance with transformer.</p>