Large errors in initial time steps while using Seq2Seq for time series prediction

Hossein_Mirzaee: <p>Hi,</p>
<p>I am trying to use a Seq2Seq model for time series prediction (prediction of some target time series based on other time series). While I managed to get overall good results, for some reason, my model has a poor prediction performance in the initial timesteps. In the figure below, you can see the predictions vs target for 5 of the output time series:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/0/50c0345e14e6f3554d731a7d54f11fba962865e4.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/50c0345e14e6f3554d731a7d54f11fba962865e4" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/0/50c0345e14e6f3554d731a7d54f11fba962865e4_2_690x261.jpeg" alt="image" data-base62-sha1="bwlZCXSJ5f1b3vvS4Um0OHyWBbS" width="690" height="261" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/0/50c0345e14e6f3554d731a7d54f11fba962865e4_2_690x261.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/5/0/50c0345e14e6f3554d731a7d54f11fba962865e4_2_1035x391.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/0/50c0345e14e6f3554d731a7d54f11fba962865e4_2_1380x522.jpeg 2x" data-dominant-color="F8F8F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2152×816 149 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I am not sure, but I suspect that I might be doing something wrong regarding hidden state initialization. Here’s the piece of code I am using for Seq2Seq.</p>
<pre><code class="lang-auto">class Encoder(nn.Module):
    def __init__(self, config):
        super(Encoder, self).__init__()

        self.input_size = config.input_size
        self.hidden_size = config.gru_hidden_size
        self.output_size = config.output_size
        self.n_layers = config.n_layers
        self.seq_len = config.seq_len
        self.config = config

        bidirectional = True
        self.lstm = nn.GRU(self.input_size, config.gru_hidden_size, config.n_layers, bidirectional=bidirectional)

    def forward(self, inputs):
        inputs = inputs.permute(1,0,2)
        output, hidden = self.lstm(inputs)
        return hidden


class Decoder(nn.Module):
    def __init__(self, config):
        super(Decoder, self).__init__()

        self.input_size = config.input_size
        self.hidden_size = config.gru_hidden_size
        self.output_size = config.output_size
        self.n_layers = config.n_layers
        self.seq_len = config.seq_len
        self.config = config

        bidirectional = True
        self.lstm = nn.GRU(self.output_size, config.gru_hidden_size, config.n_layers, bidirectional=bidirectional)
        if bidirectional:
            self.fc = nn.Linear(2*config.gru_hidden_size, config.output_size)
        else:
            self.fc = nn.Linear(config.gru_hidden_size, config.output_size)

    def forward(self, input, hidden):

        input = input.unsqueeze(0)
        output, hidden = self.lstm(input, hidden)
        output = self.fc(output)
        return output, hidden


class Seq2Seq(nn.Module):
    model_name = "Seq2Seq"
    
    def __init__(self, config):
        super(Seq2Seq, self).__init__()

        self.encoder = Encoder(config)
        self.decoder = Decoder(config)

    def forward(self, inputs, targets):

        seq_len = targets.shape[1]

        hidden = self.encoder(inputs)
        preds = torch.zeros(targets.shape)

        input = targets[:,0,:]
        preds[:,0,:] = input

        for t in range(1,seq_len):
            output, hidden = self.decoder(input, hidden)
            preds[:,t,:] = output
            input = output.squeeze(0)
            
        return preds

</code></pre>
<p>Does anyone know what I am missing?</p>