Encoder Decoder Pytorch Predictions not matched for Time Series

talmaashani2016: <p>Hi folks,</p>
<p>I am having a slight issue to get the prediction size for time series sequence using encoder decoder RNN.</p>
<p>Here is my class Encoder Decoder:</p>
<pre><code class="lang-auto">class Encoder(nn.Module):

    def __init__(self, input_size, hidden_size, num_layers = 1):
        super(Encoder, self).__init__()
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.num_layers = num_layers

        self.gru = nn.GRU(input_size = input_size, hidden_size = hidden_size, num_layers = num_layers)

    def forward(self, x):
        flat = x.view(x.shape[0], x.shape[1], self.input_size)
        out, h = self.gru(flat)
        return out, h


class Decoder(nn.Module):

    def __init__(self, input_size, hidden_size, hidden_dl_size, output_size = 1, num_layers = 1):
        super(Decoder, self).__init__()
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        self.output_size = output_size

        self.gru = nn.GRU(input_size = input_size, hidden_size = hidden_size, num_layers = num_layers)
        self.lin1 = nn.Linear(hidden_size, hidden_dl_size)
        self.lin2 = nn.Linear(hidden_dl_size, output_size)

    def forward(self, x, h):
        out, h = self.gru(x.unsqueeze(0), h)
        y = torch.relu(self.lin1(out.squeeze(0)))
        y = self.lin2(y)
        return y, h


class EncoderDecoder(nn.Module):

    def __init__(self, hidden_size, hidden_dl_size, input_size = 1, output_size = 1):
        super(EncoderDecoder, self).__init__()
        self.input_size = input_size
        self.hidden_size = hidden_size

        self.encoder = Encoder(input_size = input_size, hidden_size = hidden_size)
        self.decoder = Decoder(input_size = output_size, hidden_size = hidden_size,
                               hidden_dl_size = hidden_dl_size, output_size = output_size)

    def train_model(
            self, train, target,
            epochs, target_len, method = 'recursive',
            tfr = 0.5, lr = 0.01, dynamic_tf = False
    ):
        losses = np.full(epochs, np.nan)
        optimizer = optim.Adam(self.parameters(), lr = lr)
        criterion = nn.MSELoss()

        best_val_loss = 1000_000
        best_model_params = None

        for e in range(epochs):
            predicted = torch.zeros(target_len, train.shape[1], 1)
            optimizer.zero_grad()
            _, enc_h = self.encoder(train)

            dec_in = train[-1, :, 0].unsqueeze(1)
            dec_h = enc_h

            if method == 'recursive':
                for t in range(target_len):
                    dec_out, dec_h = self.decoder(dec_in, dec_h)
                    predicted[t] = dec_out
                    dec_in = dec_out

            if method == 'teacher_forcing':
                # use teacher forcing
                if random.random() &lt; tfr:
                    for t in range(target_len):
                        dec_out, dec_h = self.decoder(dec_in, dec_h)
                        predicted[t] = dec_out
                        dec_in = target[t, :].unsqueeze(1)
                # predict recursively
                else:
                    for t in range(target_len):
                        dec_out, dec_h = self.decoder(dec_in, dec_h)
                        predicted[t] = dec_out
                        dec_in = dec_out

            if method == 'mixed_teacher_forcing':
                # predict using mixed teacher forcing
                for t in range(target_len):
                    dec_out, dec_h = self.decoder(dec_in, dec_h)
                    predicted[t] = dec_out
                    # predict with teacher forcing
                    if random.random() &lt; tfr:
                        dec_in = target[t, :].unsqueeze(1)
                    # predict recursively
                    else:
                        dec_in = dec_out

            loss = criterion(predicted.squeeze(2), target)
            loss.backward()
            optimizer.step()

            if e % 10 == 0:
                print(f'Epoch {e}/{epochs}| 'f'test: {round(loss.item(), 4)}')

            # dynamic teacher forcing
            if dynamic_tf and tfr &gt; 0:
                tfr = tfr - 0.02

        return best_model_params, best_val_loss

    def predict(self, x, target_len):
        y = torch.zeros(target_len, x.shape[1], 1)
        print(f"Y Shape {y.shape}")
        _, enc_h = self.encoder(x)
        dec_in = y[-1, :, 0].unsqueeze(1)
        print(f"Decoder input {dec_in[0].shape}")
        dec_h = enc_h

        for t in range(target_len):
            dec_out, dec_h = self.decoder(dec_in, dec_h)
            y[t] = dec_out
            dec_in = dec_out

        return y
</code></pre>
<p>it has all the functions.</p>
<p>My issue is with y dimension that it not matching the size of y_test_tensor:</p>
<pre><code class="lang-auto"> torch.Size([90, 1]))
</code></pre>
<p>and if I looked at output of predcition method it is [90,10,1] and 10 is my batch size.</p>
<p>Can you support.</p>
<p>Thanks</p>

talmaashani2016: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> please support</p>

ptrblck: <p><code>nn.GRU</code> uses <code>batch_first=False</code> by default and thus expects an input in the shape <code>[seq, batch_size, nb_features]</code> as described in the <a href="https://pytorch.org/docs/stable/generated/torch.nn.GRU.html">docs</a>.</p>
<p>In your code you call <code>(un)squeeze(0)</code>  thus adding and removing the sequence dimension.<br>
Did you check this is indeed the desired bahevior? It’s often easier to use <code>batch_first=True</code> and keep the batch dimension in <code>dim0</code> as the majority of the layers expect inputs in this shape.</p>

talmaashani2016: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a> for your response, I have amend the changes but I am still receiving the issue.</p>
<p>can you recommended a tutorial for implementing this in pytorch.</p>
<p>I am trying to utilize a set of features to predict one column of a time series using the above.</p>
<p>Thanks for yor ultimate support as usual</p>