How to design a decoder for time series regression in Transformer?

JiangtaoLiu: <p>I am using Transformer for time series regression (not forecasting). My input data has the structure [batch, seq_size, embedding_dim], and my output structure is [batch, seq_size, 1]. But the result of the model is always overfitting and is worse than the LSTM. I don’t want to include the target information in the decoder. Can anyone tell me how to design the decoder?</p>
<pre><code class="lang-auto">import torch.nn as nn

class Transformer(nn.Module):

    def __init__(self, d_input, n_head, d_model, n_layer):
        super(Transformer, self).__init__()

        self.enc_position_embedding = self.data_position_embedding(c_in=d_input, d_model=d_model)
        self.encoder_layer = nn.TransformerEncoderLayer(d_model=d_model,  nhead=n_head)
        self.encoder = nn.TransformerEncoder(self.encoder_layer, num_layers=n_layer)

        self.project = nn.Linear(d_model, 1)

    def forward(self, x):

        x_emb = self.enc_embedding(x)
        enc = self.encoder(x_emb)

        dec = enc

        output = self.project(dec)
        return output
</code></pre>
<p>Structure:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/6/96fb9fdefa4b28900c0ffd86d2cd8b1e271a15de.png" data-download-href="https://discuss.pytorch.org/uploads/default/96fb9fdefa4b28900c0ffd86d2cd8b1e271a15de" title="Image 4"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/6/96fb9fdefa4b28900c0ffd86d2cd8b1e271a15de_2_354x499.png" alt="Image 4" data-base62-sha1="lxEIRnysBleiNVk1Blz9DCQAyMm" width="354" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/6/96fb9fdefa4b28900c0ffd86d2cd8b1e271a15de_2_354x499.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/6/96fb9fdefa4b28900c0ffd86d2cd8b1e271a15de_2_531x748.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/6/96fb9fdefa4b28900c0ffd86d2cd8b1e271a15de_2_708x998.png 2x" data-dominant-color="DEE1E0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Image 4</span><span class="informations">1320×1860 170 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>loss curve:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/f/bf30b2cad932fb41a270eb0260daa0c91fa9bd72.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/bf30b2cad932fb41a270eb0260daa0c91fa9bd72" title="loss"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/f/bf30b2cad932fb41a270eb0260daa0c91fa9bd72_2_319x250.jpeg" alt="loss" data-base62-sha1="rhlx1Ks3KWtoZWlGb6HV5Xf3Y9s" width="319" height="250" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/f/bf30b2cad932fb41a270eb0260daa0c91fa9bd72_2_319x250.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/b/f/bf30b2cad932fb41a270eb0260daa0c91fa9bd72_2_478x375.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/f/bf30b2cad932fb41a270eb0260daa0c91fa9bd72_2_638x500.jpeg 2x" data-dominant-color="F8F8F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">loss</span><span class="informations">3457×2703 258 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

WowPy: <p>Hello,</p>
<p>I have a similar scenario. Did you manage to solve the issue?</p>

basilique: <p>I have the exact same problem. Did you find out how this issue should be addressed? Thank you.</p>

Data_Curious: <p>I just came across this post.<br>
Why are you using both encoder and decoder? If you’re not feeding decoder output (“Output Probabilities”, or in your case, output from the preceding Linear layer), I don’t think you need the decoder. You are not using cross-attention. You’ve got the what are you “Output embedding” block crossed out… So, it looks like you’re not feeding anything into the Masked Multihead Attention block.</p>
<p>You should try using the encode only. Instead of feeding the encoder Add &amp; Norm output into a decoder, just feed it into a Linear layer for your output (and then, possible, a ReLU or tanh layer, depending on the normalization of your data).</p>