pytorch sequence to sequence modelling via encoder decoder for time series

username: seq2seq1
cooked: <p>I was following the the tutorial on PyTorch website for seq2seq modelling, following are the parts of the code I am using:</p>
<pre><code>class EncoderRNN(nn.Module):
    def __init__(self, input_size, hidden_size):
        super(EncoderRNN, self).__init__()
        self.hidden_size = hidden_size

        self.embedding = nn.Embedding(input_size, hidden_size)
        self.gru = nn.GRU(hidden_size, hidden_size)

    def forward(self, input, hidden):
        embedded = self.embedding(input).view(1, 1, -1)
        output = embedded
        output, hidden = self.gru(output, hidden)
        return output, hidden

    def initHidden(self):
        return torch.zeros(1, 1, self.hidden_size, device=device)

class AttnDecoderRNN(nn.Module):
    def __init__(self, hidden_size, output_size, dropout_p=0.1, max_length=MAX_LENGTH):
        super(AttnDecoderRNN, self).__init__()
        self.hidden_size = hidden_size
        self.output_size = output_size
        self.dropout_p = dropout_p
        self.max_length = max_length

        self.embedding = nn.Embedding(self.output_size, self.hidden_size)
        self.attn = nn.Linear(self.hidden_size * 2, self.max_length)
        self.attn_combine = nn.Linear(self.hidden_size * 2, self.hidden_size)
        self.dropout = nn.Dropout(self.dropout_p)
        self.gru = nn.GRU(self.hidden_size, self.hidden_size)
        self.out = nn.Linear(self.hidden_size, self.output_size)

    def forward(self, input, hidden, encoder_outputs):
        embedded = self.embedding(input).view(1, 1, -1)
        embedded = self.dropout(embedded)

        attn_weights = F.softmax(
            self.attn(torch.cat((embedded[0], hidden[0]), 1)), dim=1)
        attn_applied = torch.bmm(attn_weights.unsqueeze(0),
                                 encoder_outputs.unsqueeze(0))

        output = torch.cat((embedded[0], attn_applied[0]), 1)
        output = self.attn_combine(output).unsqueeze(0)

        output = F.relu(output)
        output, hidden = self.gru(output, hidden)

        output = F.log_softmax(self.out(output[0]), dim=1)
        return output, hidden, attn_weights

    def initHidden(self):
        return torch.zeros(1, 1, self.hidden_size, device=device)
</code></pre>
<p>Once the encoder and decoder have been defined, the training proceeds as follows:</p>
<pre><code>def train(input_tensor, target_tensor, encoder, decoder, encoder_optimizer, decoder_optimizer, criterion, max_length=MAX_LENGTH):
    encoder_hidden = encoder.initHidden()

    encoder_optimizer.zero_grad()
    decoder_optimizer.zero_grad()

    input_length = input_tensor.size(0)
    target_length = target_tensor.size(0)

    encoder_outputs = torch.zeros(max_length, encoder.hidden_size, device=device)

    loss = 0
    for ei in range(input_length):
        encoder_output, encoder_hidden = encoder(input_tensor[ei], encoder_hidden)
        encoder_outputs[ei] = encoder_output[0, 0]

    decoder_input = torch.tensor([[SOS_token]], device=device)

    decoder_hidden = encoder_hidden

    for di in range(target_length):
        decoder_output, decoder_hidden, decoder_attention = decoder(decoder_input, decoder_hidden, encoder_outputs)

        loss += criterion(decoder_output, target_tensor[di])
        decoder_input = target_tensor[di]  # Teacher forcing

    loss.backward()

    encoder_optimizer.step()
    decoder_optimizer.step()

    return loss.item() / target_length

def trainIters(encoder, decoder, n_iters, print_every=1000, plot_every=100, learning_rate=0.01):
    encoder_optimizer = optim.SGD(encoder.parameters(), lr=learning_rate)
    decoder_optimizer = optim.SGD(decoder.parameters(), lr=learning_rate)
    
    criterion = nn.NLLLoss()

    for iter in range(1, n_iters + 1):
        input_tensor = torch.from_numpy(np.array([np.random.randint(2) for i in range(10)]).reshape(10,1))
        target_tensor = torch.from_numpy(np.array([np.random.randint(2) for i in range(10)]).reshape(10,1))
                
        loss = train(input_tensor, target_tensor, encoder, decoder, encoder_optimizer, decoder_optimizer, criterion)
        print("loss===&gt;",loss)

        
hidden_size = 256
encoder1 = EncoderRNN(50, hidden_size).to(device)
attn_decoder1 = AttnDecoderRNN(hidden_size, 50).to(device)

trainIters(encoder1, attn_decoder1, 10000, print_every=5000)
</code></pre>
<p>This code works for translation of sentences. It picks a word, generates its embedding and then gets a hidden state via a GRU in the encoder. This hidden state is then passed to the decoder word by word and for each word there is a vector of probabilities. Then the Negative log likelihood loss is computed given the target label and vector of probabilities.</p>
<p>My question is basically how to adapt this to a time series forecasting model? I have a time series data divided into two parts, sequence 1 and 2. I wish to predict sequence 2. It is clear to me that I need the MSE Loss instead of the classification loss. Also, I believe there is no need to generate embeddings for a particular value in the time series. Could someone experienced with PyTorch please explain how to modify this?</p>

username: Gautham_Kumaran
cooked: <p>Almost 2 years later, I’m trying to achieve the same <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=12" title=":sweat_smile:" class="emoji" alt=":sweat_smile:" loading="lazy" width="20" height="20"> here is my implementation</p><aside class="quote quote-modified" data-post="1" data-topic="80463">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gautham_kumaran/48/16948_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/seq2seq-model-with-attention-for-time-series-forecasting/80463">Seq2seq model with attention for time series forecasting</a> 
  </div>
  <blockquote>
    Hi, I’m putting together a basic seq2seq model with attention for time series forecasting. I can’t find any basic guide to achieve this, so I’m following this NLP tutorial. (<a href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html</a>) and trying to convert it to time series forecasting. 
This is the time series model architecture I have now. 
 <a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/0/004440b0b16d692b2171fe564896e0eafb5ddfda.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/004440b0b16d692b2171fe564896e0eafb5ddfda" title="enc dec raw">[enc dec raw]</a> 
I have followed the architecture in the tutorial and removed all the embedding layers needed for NLP. 
Following is the co…
  </blockquote>
</aside>